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ABSTRACT 

During  the  past  decade  there  has  been  increasing 
public  and  political  acceptance  of  the  utility  of  inquiry  and 
inquiry- related  activities  for  bringing  about  educational 
improvement.  Specifically,  educational  research,  development, 
diffusion,  and  evaluation  (RDDE)   iave  been  singled  out  as  essential 
activities.  This  document  presents  the  report  of  a  study  designed: 
(1)  to  develop  a  conceptual  map  representative  of  the  functions 
(roles) ,  tasks,  and  skills  of  educational  researchers  and 
research-related  personnel;   (2)   to  utilize  the  data  generated  in 
objective  1  in  dcsigiing  and  developing  multidimensional  competence 
instruments  and  assessment  procedures  to  ascertain  the  extent  to 
which  resissarch  and  research-related  personnel  possess  the  identified 
competencies;    (3)  to  inventory  and  describe  current  training  programs 
in  educational  RRDE  and  their  human  and  materials  products;   (4)  to 
determine  how  many  persons  are  needed  to  perform  each  function 
identified  in  objective  1,  i.e.,  what  is  the  demand  for  trained 
personnel  to  perform  eac.Vi  function;  (5)  to  revise  existing  manpower 
projections  based  on  recent  funding  of  RRDE  activities;  and  (6)  to 
synthesize  existing  manpower  studies  based  on  nonfinancia] 
projections.  (Author/KS) 
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INTRODUCTION 

During  the  past  decade  there  has  been  increasing  public  and  politi- 
cal acceptance  of  the  utility  of  inquiry  and  inquiry-related  activities  for 
bringing  about  educational  improvements.    Specifically,  educational  research, 
development,  diffusion,  and  evaluation  (RDDE)  have  been  singled  out  as  essen- 
tial activities.    Financial  support  for  all  these  activities  increased 
dramatically  during  the  1960s  and  recent  trends  toward  educational  account- 
ability have  given  added  impetus  to  educational  evaluation.   The  proposed 
National  Institute  of  Education  will  place  additional  emphasis  on  research 
activities  and  on  the  construction  of  an  educational  knowledge  base  for  the 
future. 

Initially,  the  rapid  accretion  of  RDDE  activities  demanded  substantial 
increases  in  the  numbers  of  qualified  personnel  to  assume  the  many  new  posi- 
tions created  by  new  RDDE  undertakings.    This  manpower  shortage  had  at 
least  two  results:    (1)  new  programs  were  initiated  to  train  more  RDDE  per- 
sonnel, and  (2)  in  the  absence  of  sufficient  nunters  of  persons  specifically 
trained  in  RDDE,  other  persons  were  pressed  into  service  in  RDDE  positions 
with  the  hope  or  expectation  that  they  could  become  sufficiently  trained 
on  the  job  to  function  effectively  in  RDDE  activities.    Employment  opportuni- 
ties for  the  relatively  small  cadre  of  persons  trained  specifically  in  RDDE 
were  many. 

• 

More  recently,  financial  support  for  educational  RDDE  has  leveled 
off  and,  in  many  areas,  has  been  reduced  sharply.    Many  RDDE  programs  have 
been  discontinued,  positions  eliminated,  and  RDDE  personnel  left  to  seek 
new  positions.   General  deterioration  on  the  economic  scene  has  coupled 
with  RDDE  fund  shortages  lo  create  a  situation  In  which  the  number  of 
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applicants  per  position  opening  has  incre«ised  in  many  areas  and  graduates 
of  RDDE  training  programs  have  had  an  increasingly  difficult  time  finding 
suitable  positions.   This  has  led  many  observers  to  the  opinion  that  the 
market  for  RDDE  personnel  in  education  is,  at  least  for  the  moment,  glutted 
and  "pre-service"  training  of  such  personnel  should  be  de-emphasized.  Others, 
however,  have  argued  that  it  would  be  most  unfortunate  to  allow  temporary 
reductions  in  RDDE  funds  to  direct  energy  and  resources  away  from  pre- 
service  RDDE  training  and  thus  lose  valuable  lead  time  needed  to  prepare 
persons  for  new  positions  which  will  doubtlessly  arise  as  new  inquiry  and 
inquiry-related  programs  are  initiated  in  the  field  of  education.  Univer- 
sities and  colleges  continue  to  develop  new  programs  to  prepare  educational 
researchers,  evaluators,  product  developers,  information  spe* allots,  and 
the  like. 

Even  if  the  arguments  for  de-emphasizing  pre-service  RDDE  training 
prevail,  however,  such  a  "breathing  spell"  only  allows  a  related  problem 
to  come  into  focus  more  clearly.    It  is  becoming  increasingly  apparent 
through  both  casual  observation  and  empirical  investigations^  that  mar\y 
educational  RDDE  role  occupants  are  inadequately  prepared  for  the  positions 
they  now  hold.    It  is  clear  that  many  per*sons  performing  in  RDDE  roles  are 
sever-ely  handicapped  by  their  lack  of  skill  and/or  knowledge  in  substantive 
and  methodological  requisites  in  their  areas  of  investigation.    It  is  equally 
clear  that  even  the  RDDE  personnel  who  are  initially  well-trained  to  engage 
in  their  respective  activities  are  faced  with  obsolescence  in  a  fieJd  almost 
devoid  of  viable  programs  designed  to  keep  such  persons  abreast  of  new 
developments  in  their  areas  of  specialization.   The  problem  of  obsolescence 

^See,  for  example,  Worthen  ano  ?yers  (1970). 
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is  especially  acute  in  methodologies  and  technological  developments  in 
skill  areas  such  as  statistics  (e.g.,  Uayesian  theory)  and  computer  tech- 
nology which  are  essential  tools  in  educational  inquiry.    Overall,  there 
appear  to  be  serious  deficiencies  in  the  initial  preparation  or  updating 
of  substantial  numbers  of  RDDE  workers,  pointing  to  a  need  for  increased 
in-service  training  efforts. 

The  American  Educational  Research  Association  (AERA)  has  attenpted 
to  meet  the  demand  for  in-service  RDDE  training  by  conducting  during  the 
past  several  years  a  number  of  Presessions  of  educational  research  train- 
ing, each  of  several  days'  duration,  prior  to  its  Annual  Meeting.  The 
large  number  of  persons  who  apply  annually  for  admission  into  these  sessions 
(and  during  the  last  year  paid  a  significant  registration  fee)  suggests  that 
educational  researchers  themselves  recognize  their  need  for  training  in 
new  methods  and  techniques  and  are  active  in  seeking  such  training  oppor- 
tunities. 

The  development  of  high-quality  RDDE  training  programs-whether  pre- 
service  or  in-service-depends  on  careful  planning  of  the  substantive  and 
experiential  content  of  the  programs.   Lack  of  knowledge  about  "training 
variables,"  however,  is  undoubtedly  the  greatest  impediment  to  planning 
training  programs  which  will  not  only  provide  sufficient  initial  training 
in  RDDE  but  also  provide  sufficient  inservice  training  to  prevent  obsoles- 
cence and  continually  upgrade  skills.  Definitive  information  is  needed  on 
many  factors  relating  to  RDDE  training.    For  example,  there  is  a  pressing 
need  to  identify  the  functions  and  tasks  RDDE  personnel  must  perform  and 
the  particular  skills  and  knowledge  necessary  to  permit  effective  performance 
of  these  functions  and  tasks.  There  is  also  a  critical  need  for  apprt)aches 
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and  techniques  for  assessing  the  competencies  (skills  and  knowledge)  of 
RDDE  personnel  and  trainees  in  education.    Another  need  is  to  ascertain  as 
accurately  as  possible  the  present  parameters  of  the  educational  RDDE 
community  and  identify  the  training  programs  and  opportunities— both  pre- 
service  and  inservice— which  exist  therein.    It  is  also  necessary  to  try 
to  determine  how  many  new  positions  in  RDDE  will  actually  be  available  in 
the  next  few  years.   A  related  need  is  for  a  system  wMch  will  monitor 
accurately  the  cnanging  status  of  the  entire  educational  research  milieu 
as  it  relates  to  personnel,  their  training,  and  the  competencies.    It  is 
also  necessary  to  determine  whether  the  results  of  existing  studies  relating 
to  RDDE  training  can  be  synthesized  to  derive  generalizations  more  useful 
to  RDDE  trainers  than  are  the  results  of  the  studies  when  viewed  individually. 

The  needs  listed  above  require  periods  of  study  and/or  development 
before  alleviating  activities  can  be  operationalized.  To  begin  to  establish 
a  relevant  knowledge  base  and  develop  procedures  to  attack  these  problems, 
AERA  established  a  Task  Force  on  the  Training  of  Research  and  Research- 
related  Personnel  In  Education  in  1969.   Under  support  of  a  USOE  grant,  the 
first  pilot  year  of  the  Task  Force  operation  was  devoted  to  exploring  seve^'al 
strategies  for  collecting  data  related  to  training  variables.   Efforts  were 
aimed  at  exploring  training  needs  ot  educational  RDDE  personnel,  describing 
methods  for  improving  RDDE  training  efforts,  and  planning  studies  and  develop- 
mental activities  to  satisfy  several  of  the  needs  outlined  above.  Specifi- 
cally, the  pilot  year  activities  enabled  the  AERA  Task  Force  to  delineate 
parameters  and  problems  of  RDDE  training  in  sufficient  detail  to  dictate 
specific  data  collection  and  developmental  activities  needed  to  develop 
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solutions  for  many  of  the  continuing  problems  facing  those  responsible 
for  establishing  and  conducting  programs  for  preparing  RDDE  personnel 
in  education. 

Several  of  the  needs  discussed  above  are  reflected  in  the  major 
objectives  of  the  present  study,  which  are  listed  below. 

Objectives 

The  major  objectives  ;md  r.ub-objectives  of  the  Task  Force  project 
for  1970-71  are  listed  below; ^ 

1.   To  develop  a  conceptual  map  representative  of  the  functions 
(roles),  tasks,  and  skills  of  educational  researchers  and 
research-related  personnel. 

1.1  To  develop  a  conceptual  framework  defining  and  delimiting 
the  salient  characteristics  of,  and  interrelationships 

J  between,  research,  development,  diffusion,  and  evaluation 
>    in  education. 

1.2  To  identify  and  delineate  the  functions  (roles)  that  are 
required  of  research  and  research -related  personnel  and 
modify  the  above  conceptual  framework  as  needed. 

1.3  To  identify  and  delineate  the  specific  tasks  that  are 
required  in  the  performance  of  each  function  identified  in 
objectives  1.1  and  1.2. 


2 

with  ^■hJ'ftlli      ^^'f  P"»Jfct,  the  scope  of  work  for  the  grant  was  renegotiated 
with  the  USOE,  resulting  in  deletion  of  three  objectives  (objectives  three, 
seven,  and  eight  in  the  original  proposal),  addition  of  one  sub-objective 
(4.3  In  the  present  list),  and  minor  modifications  in  others.    At  that 
time  the  Task  Force  also  refined  the  original  objectives,  resulting  in 
minor  modifications  in  wording  and  subordination.    However,  the  refinements 
did  not  cause  substantive  changes  or  procedural  modification;  these  minor 
chanp  were  introduced  only  to  increase  clarity  and  better  represent  the 
original  Intent  of  the  Task  Force  in  proposing  the  activities  reported  herein. 

thncn  4!!^J!!**"^4*f^  r*"*^  objectives  listed  above  is  not  identical  with 

those  in  the  original  proposal,  the  nunbering  of  objectives  used  here  and 
throughout  this  report  is  consistent  with  the  nunfcering  In  the  original 
proposal .  ^ 
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1.4   To  identify  and  delineate  the  competencies  (skills  and 
knowledge)  necessary  to  perfonn  each  task  identified  in 
objective  1.3. 

To  utilize  the  data  generated  in  objective  1  in  designing  and 
developing  multidimensional  competence  instruments  and  assess- 
ment procedures  to  ascertain  the  extent  to  which  research  and 
research-related  personnel  possess  the  identified  competencies. 

2.1  To  design  and  develop  one  pilot  mastery  test  of  certain 
competencies  in  RDDE, 

2.2  To  design  and  develop  questionnaires  to  assess  RDDE  stu- 
dents' and  workers'  perceptions  of  their  existing  competen- 
cies and  training  needs. 

To  inventory  and  describe  current  training  programs  in  educa- 
tional RDDE  and  their  human  and  materials  products. 

4.1  To  determine  how  many  formal  programs  exist  specifically 
for  training  RDDE  pc  «onnel  in  education  (i.e.,  what 
opportunities  exist  for  initial  training  or  retraining  of 
such  personnel). 

4.2  To  determine  the  characteristics  of  these  formal  training 
programs  in  terms  of  (a)  number  and  educational  level  of 
trainees;  (b)  duration  of  training;  (c)  financial  support; 
(d)  program  focus;  and  (e)  experiences  and  courses  pro- 
vided or  required. 

4.3  To  produce  a  compendium  including  a  brief  description  of 
each  training  program  identified  in  sub-objective  4.1  above 
on  characteristics  such  as  those  listed  in  sub-objective  4.2. 

To  determine  how  many  persons  are  needed  to  perform  each  function 
Identified  (in  objective  1);  i.e.,  what  is  the  demand  for  trained 
personnel  to  perfonn  each  function? 

5.1  To  revise  existing  manpower  projections  based  on  recent 
funding  of  RDDE  activities. 

5.2  To  synthesize  existing  manpower  studies  based  on  nonfinancial 
projections. 

To  supplement  the  information  yielded  by  attainment  of  objectives 
I  through  5  above  by  synthesizing  extant  studies  of  educational 
RDDE  personnel  or  studies  of  variables  directly  related  to  train- 
ing such  persons. 
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Overview  of  This  Report 

To  attain  the  objectives  listed  in  the  preceding  section,  it  was 
necessary  to  conduct  several  discrete  research  and  development  activities. 
Although  all  the  activities  are  bonded  together  by  a  common  feature- 
concern  with  the  production  of  planning  data  or  materials  necessary  to 
make  decisions  about  priorities  in  RDDE  training  program  development  and 
implementation —separate  activities  were  required  to  attain  each  objective 
and,  in  some  cases,  each  sub-objective.    Therefore,  this  larger  report 
contains  reports  of  seven  distinct  and  separate  (although  interrelated) 
research  and  development  activities.   Each  of  these  seven  activities  has 
been  reported  in  a  series  of  technical  papers  produced  by  the  Task  Force. ^ 
The  technical  papers  represented  Interim  reports  and  have  been  incorporated 
in  edited  form  in  this  report. 

To  orient  the  reader  to  (a)  the  relationship  between  the  objectives 
and  the  remainder  of  this  report,  and  (b)  the  relationship  of  the  technical 
papers  to  the  various  sections  of  this  report,  the  following  discussion  is 
included.    In  it,  each  objective  Is  discussed  and  the  relevant  technical 
papers  and  chapters  In  the  present  report  are  referenced. 

Objective  1 

Objective  1  and  its  subobjectlves  are  interrelated  and  were  appraoched 
as  a  group.   Procedures  relating  to  those  subobjectlves  are  discussed  herein 
in  Chapter  II  (and  were  reported  previously  in  Technical  Papers  No.  18  and  23). 

3 

A  list  of  technical  papers  by  nunter,  author(s),  and  title  is  included 
in  the  list  of  references  at  the  end  of  this  report. 
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Briefly,  Chapter  II  contains  a  discussion  of  an  effort  to  determine  empiri- 
cally the  functions,  tasks,  and  cpr.,pefcencies  required  of  RDDE  workers  through 
a  series  of  "task-analysis"  interviews  and  factor  analysis  of  the  resultant 
data. 

Objective  2 

Objective  2  comprises  two  sub-objectives  which  are  reversed  in  the 
sequence  in  which  they  are  presented  in  this  report.    Sub-objective  2.2  is 
directly  related  to  objective  1  in  that  RDDE  competencies  identified  through 
objective  1  activities  were  used  as  the  basis  for  sub-objective  2.2  instru- 
ment construction  efforts.    Procedures  relating  to  sub-objective  2.2  are 
presented  in  Chapter  III  (and  previously  were  reported  in  Technical  Paper 
No.  30).    Chapter  III  contains  a  description  of  the  development  of  an 
instrument  to  obtain  self-report  data  on  selected  competencies  from  objec- 
tive 1. 

Procedures  relating  to  subobjective  2.1  are  presented  in  Chapter  IV 
(previously  in  Technical  Paper  No.  29).  Chapter  IV  includes  a  description 
of  the  development  of  a  pilot  test  for  assessing  competencies  of  personnel 
in  educational  research,  development,  and  evaluation. 

Objective  4 

Procedures  relating  to  sub-objectives  4.1  and  4.2  are  reported  in 
Chapter  V  (previously  in  Technical  Paper  No.  24).   BHefly.  Chapter  V  con- 
tains a  description  of  a  survey  to  Identify  existing  pre-service  and  in- 
service  RDDE  training  opportunities  In  the  United  States. 
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Sub-objective  4.3  was  attained  through  taking  data  on  each  program 
identified  through  the  survey  mentioned  above  and  transferring  it  to  a 
one-page  description  of  that  program  for  inclusion  in  a  compendium  of 
training  opport unities.    The  resulting  compendium  appears  as  a  separately 
bound  appendix  and  is  not  discussed  further  in  the  body  of  this  report. 

Objective  5 

Procedures  relating  to  sub-objective  5.1  are  reported  in  Chapter  VI 
(previously  in  Technical  Paper  No.  25).   That  chapter  contains  a  discussion 
of  efforts  to  update  the  original  CI  ark -Hopkins  educational  RDD  manpower 
projections  for  1974  in  view  of  changes  in  funding  patterns  which  invali- 
dated many  of  the  assumptions  on  which  the  earlier  projections  were  based. 

Procedures  relating  to  sub-objective  5.2  are  presented  in  Chapter 
VII  (and  were  reported  previously  in  Technical  Paper  No.  26).    Chapter  VII 
presents  an  attempt  to  identify,  review,  and  synthesize  RDDE  manpower 
studies  to  try  to  identify  trends  in  supply  and  demand  of  RDDE  personnel 
In  education. 

Objective  6 

Procedures  relating  to  objective  6  are  reported  In  Chapter  VIII 
(and  previously  In  Technical  Papers  27  and  28).*  That  chapter  contains  (a) 
a  description  of  a  process  used  to  identify,  review,  and  critique  all 

technical  Paper  No.  27  contained  two  major  sections:    (a)  a  descrip- 
tion of  a  procedure  for  determinina  the  methodological  adequacy  of  each  study 
reviewed  under  objective  6,  and  (b)  descriptions  of  each  study  included  in 
the  synthesis  described  above.   The  volunlnous  second  section  is  not  repro- 
duced herein;  however.  Technical  Paper  No.  27  (referenced  later  in  this 
paper)  can  be  obtained  through  the  knic  system. 
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existing  studies  which  were  found  to  relate  to  training  of  educational 
RDDE  personnel,  and  (b)  a  synthesis  of  the  findings  of  those  studies  found 
to  be  methodologically  adequate. 

A  Further  Word  of  Orientation 

Two  further  comments  may  be  helpful  to  the  reader.    First,  there  is 
no  final  summary  or  concluding  chapter  in  this  report.   The  next  seven 
chapters  contain  reports  of  separate  research  and  development  activities 
and  each  chapter  contains  its  own  summary  or  conclusions  necessary  to  the 
report  of  that  activity.   Because  of  the  discreteness  of  the  seven  activi- 
ties reported  in  these  chapters,  providing  an  omnibus  summary  or  set  of 
conclusions  would  be  inappropriate  if  not  impossible. 

Secondly,  the  style  used  in  this  report  may  differ  somewhat  in  tone 
from  that  in  the  usual  report  of  research.    Kaplan  (1964,  pp.  3-11)  points 
out  the  differences  between  "loglc-in-use,"  the  more  or  less  logical  opera- 
tors used  by  researchers  in  conducting  their  work,  and  "reconstructed  logic," 
an  after-the-fact  idealization  of  those  operations~1  .e. ,  what  the  process 
would  be  if  extracted  and  refined  so  as  to  remove  all  alogic  or  illogic. 
Most  reports  of  research  seem  to  present  a  reconstructed  logic  of  how  the 
stu<^y  was  planned  and  conducted.   The  many  false  starts,  failures,  and 
blind  alleys  that  are  so  typical  of  the  research  process— and  so  much  a  part 
of  the  researcher's  logic-in-use— are  routinely  excised  from  final  research 
reports.  The  opposite  strategy  has  been  used  In  the  present  report.    In  the 
Interest  of  forewarning  others  who  taay  pursue  similar  Investigations  or 
developmental  efforts  of  snares  which  have  been  encountered  by  the  Task  Force 
and  staff,  the  pristine  loglc-ln-use  which  guided  each  activity  reported 
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herein  has  been  described  as  faithfully  as  possible.   To  do  so,  thus 
rendering  the  critic's  task  inorp  si-nple,  seems  preferable  to  glossing 
over  difficulties  which  were  oncount(^red  in  the  conduct  of  the  work 
reported  in  this  volume. 
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CHAPTER  II 

AN  ANALYSIS  AND  INTERPRETATION  OF  TASKS  AND  COMPETENCIES 
REQUIRED  OF  PERSONNEL  CONDUCTING  EXEMPLARY  RESEARCH 
AND  RESEARCH-RELATED  ACTIVITIES  IN  EDUCATION 
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AN  ANALYSIS  AND  INTERPRETATION  OF  TASKS  AND  COMPETENCIES 
REQUIRED  OF  PERSONNEL  CONDUCTING  EXEMPLARY  RESEARCH 
AND  RESEARCH-RELATED  ACTIVITIES  IN  EDUCATION 

• 

One  of  the  most  serious  impediments  to  efforts  to  plan  or  conduct 
training  programs  for  research  or  research-related  personnel  in  education  is 
lack  of  knowledge  about  which  particular  competencies  or  skills  are  most 
important  ijri  conducting  research  and  research- related  activities.   As  part 
of  a  continuing  effort  to  identify  competencies  that  should  be  developed  in 
training  programs,  the  Task  Force  proposed  a  series  of  site  visits  and 
interviews  designed  to  collect  data  on  specific  tasks  and  competencies  required 
of  exemplary  researchers,  evaluators,  developers,  and  diff users  in  the  field 
of  education. 

Ob jecti  ves 

Earlier  work  of  the  Task  Force  included  efforts  to  develop  a  con- 
ceptual framework  defining  and  delimiting  the  salient  characteristics  of, 
and  interrelationships  among,  research,  development,  diffusion,  and  evalua- 
tion in  education.    (See  Technical  Papers  No.  1  and  4  in  the  Task  Force 
series.)  These  conceptual  efforts  also  resulted  in  the  identification  of 
specific  skills  and  knowledge  in  each  of  these  areas  which,  on  an  a  priori 
basis,  seemed  essential  to  effective  participation  in  educational  research 
and  related  areas  (Technical  Papers  No.  1  and  5).    Enpirical  data  were  then 
collected  to  see  which  of  the  skills  were  perceived  by  employers  and  super- 
visors to  be  necessa»7  in  conducting  research,  evaluation,  development,  and 
diffusion  activities  in  education  (Technical  Papers  No.  2  and  3). 
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Although  the  above  procedures  yielded  useful  data,  they  also  brought 
to  light  three  serious  limitations.   First,  there  was  no  empirical  basis 
for  selecting  research,  evaluation,  development,  and  diffusion  as  the  general 
categories  within  which  to  classify  Inquiry  activities.   The  categories  were 
selected  because  of  their  common  use  In  the  literature  and  because  they 
seemed  Intuitively  to  encompass  most  of  the  activities  relevant  to  Inquiry 
In  education.   That  Is,  the  categories  ma^y  reflect  unknown  biases  of  the 
educational  research  community  to  a  greater  extent  than  they  accurately 
reflect  actual  classes  of  Inquiry  activities. 

Second,  the  listed  skills  also  lacked  an  empirical  base,  resulting  In 
the  same  possibility  for  bias  as  that  discussed  above. 

Third,  employers'  perceptions  (In  responding)  were  restricted  at 
least  In  part  to  the  categories  and  skills  provided  to  them  as  referents. 
Although  open-ended  questions  were  posed  to  provide  opportunities  for  addi- 
tional responses.  It  seems  likely  that  by  using  research,  development, 
diffusion,  and  evaluation  (RDDE)  as  organizing  rubrics  In  collecting  data, 
employers  likely  were  restricted  In  some  unknown  degree  from  responding 
beyond  the  framework  Imposed  on  their  thinking. 

The  procedures  described  here    were  designed  to  collect  data  not 
subject  to  the  above  limitations.   Specifically,  It  was  thought  that  the  most 
defensible  approach  would  be  to  conduct  task  analyses  of  what  persons  engaged 
In  educational  Inquiry  activities  actually  did  on  their  Jobs.   By  aggregating 
data  from  numerous  task  analyses  and  analyzing  these  data  It  was  hoped  that 
clusters  of  competencies  and  tasks  might  emerge  that  would  serve  to  delineate 
cTearly  the  major  classes  of  research  and  research-related  activities.  By 
not  Imposing  the  a  priori  classification  of  RDDE  on  the  data.  It  was  thought 
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that  analyses  of  the  data  might  serve  to  (a)  verify  the  RDDE  classifica- 
tion (if  four  such  clusters  emerged  from  the  data),  or  (b)  identify  new 
clusters  of  competencies  and  tasks  which  would  define  inquiry  functions 
other  than  RDDE.    Consequently,  an  "open,"  unbiased  methodology  was  evolved 
to  meet  the  following  objectives  (excerpted  from  the  objectives  listed 
earlier  in  this  report): 

1.1  To  develop  a  conceptual  framework  defining  and  delimiting  the 
salient  characteristics  of,  anH  interrelationships  among, 
research,  development,  diffus.  ji,  and  evaluation  in  education. 

1.2  To  identify  and  delineate  the  functions  (roles)  that  are  re- 
quired of  research  and  research-related  personnel  .  .  . 

1.3  To  identify  and  delineate  the  specific  tasks  that  are  required 
in  the  performance  of  each  function  identified  ...  (in  Objec- 

V6  1*2/ 

1.4  To  identify  and  delineate  the  competencies  (skills  and  knowledge) 
necessary  to  perform  each  task  identified  in  Objective  1.3. 

Work  on  Objective  1.1  was  reported  earlier  in  Technical  Paper  No.  4 
in  the  Task  Force  series  (Glass  and  Worthen,  1970),  which  also  appears  as 
Appendix  F  in  a  previous  OE  report  (Worthen  and  Byers,  1970).    Work  on 
Objectives  1.2,  1.3,  and  1.4  is  reported  in  the  remainder  of  this  chapter. 

Uses  of  the  Data.   The  data  acquired  in  this  study  have  a  variety  of 
uses,  S0II19  of  which  are  listed  below: 

1.   The  data  will  be  of  value  to  the  U.S.  Office  of  Education  and  other 
funding  agencies  in  more  precisely  identifying  (a)  the  inquiry  functions  (roles) 
actually  required  in  the  field  of  education,  (b)  the  tasks  which  must  be  per- 
formed in  relation  to  each  function,  and  (c)  the  competencies  (skills  and 
knowledge)  essential  to  conduct  each  task.   This  information  will  provide  a 
basis  for  making  more  informed  decisions  concerning  the  type  of  training  pro- 
grams that  should  be  funded  than  has  been  possible  in  the  absence  ^f  such  data. 
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2.  Descriptions  of  essential  competencies  will    be  of  value  to 
directors  of  training  programs  in  determining  what  knowledge  and  skills 
should  be  included  in  the  content  of  their  training  programs. 

3.  Descriptions  of  essential  competencies  will    be  useful  to 
AFRA  officers  in  determining  what  knowledge  and  skills  should  be  included 
in  the  AERA  presessions  and  in-service  training  programs. 

4.  Employers  who  have  identified  the  tasks  required  of  prospective 
employees  but  are  unsure  of  the  competencies  required  in  each  of  these  tasks 
can  use  these  data  as  a  basis  for  attempting  to  determine  if  a  prospective 
employee  has  the  competencies  necessary  to  perform  adequately  on  the  job. 

5.  The  data  acquired  through  procedures  described  herein  will  serve 
as  baseline  data  necessary  to  the  attainment  of  several  subsequent  objectives 
in  the  Task  Force  project. 

Definition  of  Terms 

Before  discussing  procedures  designed  to  attain  the  above  objectives, 
it  is  necessary  to  define  the  terms  "competencies,"  "tasks,"  and  "functions" 
as  used  herein. 

"Function"  is  used  as  a  descriptor  for  a  broad  range  of 
activities  or  tasks  which  taken  together  lead  to  the  attainment  of  a  par- 
ticular Inquiry  goal  (e.g.,  to  produce  generalizable  knowledge  about 
educational  phenomena).    Examples  of  function     are  research, 
development,  diffusion   and  evaluation,  although  these  four  functions  are 
not  taken  as  givens  for  this  study;  the  task  analyses  may  identify  tasks 
iKhlch  are  found  to  group  together  in  ways  that  require  different  function 
labels. 
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"Tasks"  are  specific  activities  which  are  essential  in  the  conduct 
of  each  function.   Examples  of  tasks  are  defining  a  research  problem. 


a  film,  or  analyzing  data. 

"Competencies"  are  the  specific  knowledge  and  skills  used  to  engage 
successfully  in  each  task.   For  example,  a  data  analysis  task  might  require 
(among  others)  the  fbl lowing  competencies:   knowledge  of  analysis  of 
variance,  knowledge  of  computer  operations,  the  ability  to  select  the  appro- 
priate statistical  technique,  the  ability  to  select  the  appropriate  computer 
program,  and  the  ability  to  interpret  the  results  of  an  analysis  of  variance. 

To  a  large  extent,  the  three  words  "function,"  "task,"  and 
"competency"  sirnply  express  levels  of  specificity  rather  than  different 
basic  ideas.   A  function  is  basically  a  collection  of  tasks,  as  a  task  is 
composed   of  a  number  of  competencies,  in  that  a  task  can  be  described  by  the 
specific  competencies  of  which  it  is  composed. 

Development  of  a  Data  Collection  Technique 

At  the  time  the  Task  Force  proposal  was  written,  it  was  not  known  if 
appropriate  instrumentation  existed  to  collect  the  type  of  data  specified 
above.    Therefore  it  was  necessary  to  use  a  variety  of  information  sources 
to  attempt  to  identify  extant  instrumentation  adequate  for  the  purposes 
described*  The  results  of  that  infomation  search  are  summarized 
briefly  below. 

"State  of  the  Art"  of  Relevant  Instrumentation 

■  ^  ■  ' 

Task  analytic  approaches  seemed  most  likely  to  be  useful  in  collecting 
data  required  to  meet  the  objectives  listed  above.  Standard 


writing  a  final  report  of  an  evaluation  project,  preparing  a  script  for 
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library  search  procedures  and  standard  information  services  (e.g.,  ERIC,  SIE, 

DATRIX)  were  used  to  identify  information  on  task  analysis,  job  analysis,  ^ 

employee  performance  rating,  etc.    Numerous  references  were  identified  and 

read  and  work  that  appeared  relevant  was  followed  up.  by  letters  or  telephone  1 

calls  to  authors  to  (a)  learn  of  any  new  work  in  the  field,  and  (b)  obtain  -i 

opinions  of  experts  as  to  what  data  collection  technique  would  most  suit 

our  needs.    No  attempt  is  made  here  to  review  all  the  literature  perused  ^ 

by  the  project  staff  or  to  describe  all  contacts  with  experts  in  task  and 

job  analysis.   This  section  contains  a  brief  summary  of  general  threads  of  ] 

the  search  and  a  few  representative  studies. 

A  management  orientation  that  emphasizes  performance  rating,  wage 
scales,  and  man-machine  efficiency  pervades  most  of  the  literature  on  task  ~j 
analysis.   Task  analysis  for  industrial  workers,  including  technicians, 
has  been  fairly  well  developed  while  the  difficulty  of  such  analysis  of  the  | 
work  of  professionals  is  indicated  by  its  lack  of  availability. 

Of  the  various  approaches  reviewed,  the  eclectic  Air  Force  technique,  -» 
(Morsh,  1961)  based  on  the  administration  of  duty  and  task  inventories,  is  "j 
the  most  complex.    Essentially  it  consists  of  (a)  administering  a  checklist 
with  tasks  listed  under  each  duty  and  a  provision  for  adding  tasks  not  already  liste«^ 
and  (b)  obtaining  ratings  on  two  items  —  time  spent  and  training  and  experience 
required.    A  typical  approach  (Morsh,  1963)  involves  extensive  development  -1 
work,  including  a  tryout  with  as  many  as  100  or  more  workers,  before  adminis-  ^ 
tering  the  inventories  to  large  numbers  of  people  in  Air  Force  installations 
around  the  world.    The  Air  Force  and  other  government  agencies  actively  employ  ?| 
this  approach  in  current  activities  (e.g.,  Christal ,  1969;  Brumback,  1969; 
Mayo,  1968  and  McCormick,  1969)  and  the  emphasis  remains  on  evaluating  ll 
job  performance  with  the  intent  of  assisting  management  of  operations.  g 
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Except  for  the  task  analysis  research  being  done  by  the  Air  Force, 
little  has  peen  published  recently.   Among  the  older  sources  are  various 
industry-oriented  studies  such  as  Plant  (1946)  in  which  a  questionnaire  was 
used  to  elicit  self-reports  from  sales  and  clerical  job  incumbents,  ftore 
recently.  Miller  (in  Gagne,  1962)  distinguishes  between  task  description  and 
task  analysis;  description  details  requirements  of  the  job  while  analysis 
refers  to  the  skills  required.     His  approach  is  within  a  man -machine  system 
framework  that  is  conducive  to  the  development  of  minutely  detailed  analyses 
including  the  timing  and  frequency  of  a  task.   A  somewhat  different  purpose 
is  served  by  McCormick  and  Palmer's  factor  analytic  studies  (Palmer  and 
McCormick,  1969)  of  checklist  data  relating  to  worker  activity  dimensions. 
They  concluded  that  various  human  work  activities  may  be  organized  with 
greater  simplicity  and  economy  in  terms  of  a  smaller  number  of  relatively 
independent  dimensions.    A  more  recent  approach,  listed  in  a  casebook  of 
industrial  job  analysis  procedures  (Dalton,  1970),  has  a  job-requirement 
orientation  with  the  emphasis  on  efficient  allocation  of  personnel  to 
tasks. 

Unfortunately,  none  of  the  above  lines  of  investigation  proved  to 
have  much  utility  for  the  present  study,  either  in  the  opinion  of  the  Task 
Force  staff  or  in  that  of  the  task  analysis  experts  who  were  consulted. 
The  consensus  of  both  groups  was  that  instrumentation  tailored  to  produce  the 
type  of  data  needed  would  have  to  be  developed  by  the  Task  Force. 

One  Relevant  Approach  to  Instrumentation.    One  relevant  data-collection 
methodology  not  identified  through  the  seardn  procedures  outlined  above  is. 
that  developed  by  the  Oregon  State  Department  Teaching  Research  Division  In 
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conjunction  with  their  OE-sponsored  project  on  research  training.  That 
methodology  was  reviewed  by  Task  Force  staff  as  part  of  a  continuing  liaison 
between  this    project  and  the  Oregon  study.    The  Oregon  methodology  was  found 
to  be  aimed  at  the  collection  of  similar  data,  but  there  were  several 
ways  in  which  the  Oregon  focus  differed  from  that  of  the  Task  Force  project/ 
Most  of  the  differences  were  attributable  to  either  (a)  the  more  extensive 
financial  support  for  data-collection  on  the  Oregon  study  than  was  available 
for  the  Task  Force  study,   or  (b)  the  relatively, less  comprehensive  focus 
of  the  Task  Force  stucly.    For  example,  the  Oregon  study  included  repeated 
visits  to  data-collection  sites,  the  maintenance  of  daily  logs  by  site 
employees  for  some  time  before  the  visit  of  the  data-collection  team,  etc. 
Through  such  techniques,  the  Oregon  group  intended  to  investigate  the  full 
scope  of  inquiry  activities  on  a  specific  project  at  each  selected  site, 
resulting  in  a  complete  case  stucty  of  each  project.   The  case  study 
would  contain  data  on  how  personnel  interact  with  one  another,  the  nature  of 
the  project  setting,  and  the  administrative  procedures  followed.  The  inves- 
tigation of  sensitivities  (what  may  be  termed  "affective  competencies")  which 
are  part  of  RDDE  work  was  an  additional  dimension  of  the  Oregon  study.  The 
methodology  employed  interviews  that  were  both  intensive  and  lengthy,  with 
two  major  interviews  of  several  hours  being  conducted  (on  different  days) 
with  each  interviewee. 

The  financial  constraints  on  the  Task  Force  project  did  not  permit 
the  use  of  a  technique  that  required  as  many  human  and  material  resources  as 
that  employed  on  the  Oregon  Study.    In  addition,  the  Task  Force  objectives 
were  not  concerned  with  sensitivities,  administrative  procedures,  personnel 
Interactions,  etc.   The  focus  was  exclusively  on  tasks  performed  by  research 
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and  research -related  personnel  and  the  skills  and  knowledge  required  to 
perform  those  tasks.   Therefore,  it  was  necessary  to  develop  for  the  Task 
Force  project  instrumentation  for  use  in  collecting  detailed  data  about  a 
relatively  small  part  of  the  overall  field  of  concern  in  the  Oregon  study. 

Development  of  the  Initial  Interview  Procedure 

The  ii)terview  procedure  developed  for  this  study  was  expected  to 
meet  the  follov^ing  criteria: 

1.  The  procedure  must  be  sensitive  enough  to  enable  interviewers 
to  obtain  complete  data  on  the  research  and  research-related  tasks,  skills 
and  competencies  of  each  interviewee. 

2.  The  procedure  must  be  reliable  in  that  the  interviewer's 
interpretation  of  the  information  provided  by  the  interviewee  will  be  consis- 
tent with  the  interpretation  of  a  different  interviewer  who  listens  to  a 
tape  recording  of  the  same  interview. 

3.  There  must  be  sufficient  standardization  of  the  procedure  to 
meet  the  interscorer  agreement  requirements  identified  above,  yet  sufficient 
open-endedness  to  permit  the  interviewee  to  provide  a  full  list  of  his  tasks 
and  competencies  in  a  free-response  manner  rather^^an  by  selecting  from  a 
predetermined  list  of  such  tasks  and  competencies. 

4.  The  data  obtained  must  be  in  a  form  that  makes  data  compilation 
and  analysis  manageable. 

In  the  development  of  such  a  data-gathering  procedure^  several 
approaches  were  considered  and  rejected,  some  after  only  a  study  of  the 
literature  and  others  after  actual  trial.    An^ong  the  approaches  considered 
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were  (a)  direct  observation  of  a  person  while  he  is  on  the  job,  using  some 
type  of  job  analysis  system,  (b)  soliciting  information  via  a  questionnaire 
to  be  administered  by  mail  and/or  in  a  personal  interview,  and  (c)  several 
forms  of  a  personal  interview. 

The  review  of  related  knowledge  and  practice  showed  that  currently 
available  job  analysis  systems  are  designed  for  rather  systematic  and  well- 
defined  jobs.   They  are  particularly  suited  to  improving  the  efficiency  of 
a  group  of  persons  by  aiding  in  the  allocation  of  tasks  to  various  people. 
Clearly  RDDE  work  does  not  fall  intvi  this  realm  of  systematic  and  well- 
defined  activities. 

Even  as  a  supplement  to  a  personal  contact  approach,  the  use  of  a 
mailed  questionnaire  was  rejected,  because  of  the  expected  low  rate  of 
response  and  the  probable  lack  of  carefully  considered  response  to  a  ques- 
tionnaire.  The  extensiveness  and  detail  of  the  information  required  in  our 
Investigation  ruled  out  this  approach. 

Early  in  the  consideration  of  possible  approaches,  it  became  clear 
that  some  type  of  personal  interview  would  be  necessary.    Among  the  various 
types  considered  were  (a)  the  critical  incident  technique,  (b)  a  highly 
structured  interview  in  which  the  interviewee  responds  to  predetermined 
categories  on  a  checklist  or  interview  schedule,  (c)  an  interview  based 
upon  a  log  of  activities  which  the  interviewee  would  be  asked  to  keep 
for  a  prior  period  of  time,  and  (d)  an  open-ended  or  unstructured  interview 
in  which  the  interviewee  responds  to  a  series  of  open-ended  questions 
designed  to  provide  data  for  a  rather  structured  data  schedule.    All  but 
one  of  these  approaches  were  rejected  for  reasons  given  below. 
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The  critical  incident  technique  could  be  used  to  solicit  information 
on  those  job  behaviors  which  led  to  exceptionally  effective  or  ineffective 
outcomes  in  important  parts  of  the  interviewee's  job.    It  was  rejected  with- 
out trial  because  of  its  failure  to  provide  a  complete  compilation  of  all 
of  the  essential  competencies  employed  by  the  interviewee.   The  use  of  an 
intensive  interview  based  exclusively  on  a  log  of  activities  which  the 
interviewee  maintained  over  a  period  such  as  the  two  weeks  prior  to  the 
interview  was  rejected  without  trial  for  two  reasons:    (1)  the  limited 
period  of  the  interviewee's  work  time  covered  and  (2)  the  probable  lack  of 
careful  maintenance  of  the  log  by  some  interviewees. 

The  consideration  of  these  various  approaches  finally  led  to  the 
choice  of  an  intensive  interview  designed  to  obtain  a  complete  description 
of  the  tasks  and  competencies  which  were  a  part  of  the  interviewee's  work 
during  a  period  of  one  year  immediately  preceding  the  interview.   The  major 
decision  remaining  was  the  degree  of  structure  which  the  interview  should 
have.    Careful  study  and  considerable  trial -and-error  work  with  various 
procedures  were  undertaken  before  the  final  decision  was  reached.  One 
approach  tried  was  based  on  a  checklist  developed  from  the  lists  of  skills 
compiled  In  earlier  Task  Force  work  (Technical  Paper  No.  3).    One  difficulty 
encountered  was  the  fact  that  those  lists  were  found  to  be  mixtures  of  both 
tasks  and  competencies  and  often  one  item  in  the  list  subsumed  several  others. 
The  approach  was  rejected  for  a  more  basic  reason,  however  —  the  fact  that  the 
list  originated  through  logical  (armchair)  analysis  of  leading  RDDE  personnel 
rather  than  through  studying  the  actual  tasks  performed  and  competencies 
required  for  RDDE  personnel  on  the  job.   It  was  decided  that  an  open-ended 
approach  should  be  employed  to  collect  data  on  actual  on-the-job  tasks 
and  competencies.   At  the  same  time,  the  data  produced  by  the  open-ended 
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technique  would  have  to  be  susceptible  to  extant  data  compilation  and 
analysis  techniques.   These  considerations  led  to  the  development  of  an 
initial  interview  procedure  —  a  procedure  based  on  a  series  of  open-ended 
questions  designed  to  provide  data  for  a  structured  data  schedule  and  a 
previously  designed  system  of  data  analysis. 

Critique,  Revision  and  Trvout 

No  attempt  is  made  here  to  describe  the  initial  interview  procedure, 
•  since  it  is  quite  similar  to  the  final  procedures  described  below. 
The  changes  between  the  initial  and  final  interview  procedures  came 
about  through  three  activities:   (a)  a  critique  of  the  initial  interview 
procedures  by  the  AERA  Task  Force  on  Research  Training,  (b)  revision  of  the 
interview  procedures  by  the  project  staff,  and  (c)  a  tryout  of  the  revised 
interview  procedures  with  twelve  persons  engaged  in  educational  research 
and  development.  The  tryout  data  indicated  that  the  revised  (final)  inter- 
view procedures  could  be  used  to  meet  the  objectives  for  which  the  instru- 
mentation was  designed. 

Description  of  the  Final  Interview  Procedures 

The  final  interview  procedure  was  designed  to  obtain  from  each 
int^vlewee  (a)  a  list  of  all  of  the  different  inquiry  or  inquiry-related 

if' 

tasks  which  he  performed  during  the  period  of  one  year  prior  to  the  inter- 
view and  (b)  a  list  of  the  competencies  required  of  him  to  perform  each  of 
these  tasks.    The  printed  materials  used  in  the  interview  included  (a)  the 
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interview  record  form  (single  sheet)  on  which  general  information  and  a 
list  of  the  year's  major  activities  were  recorded  (see  Appendix  A)  and  (b) 
the  interview  data  sheets  (multiple  copies)  for  recording  the  tasks  and 
related  competencies  which  were  part  of  each  of  the  major  areas  of  activity 
(see  Appendix  B).    A  small  portable  cassette  tape  recorder  also  was  used  by 
each  interviewer  to  record  the  interviews. 

Specific  Interview  Procedures.   After  completing  preliminary  activities, 
such  as  setting  up  the  tape  recorder,  the  interviewer  filled  out 
the  first  half  of  the  interview  record  form  which  required  descriptive  infor- 
mation such  as  names,  titles  and  dates.    The  interviewee  was  then  asked  to 
list  the  major  projects  or  areas  of  activity  in  which  he  engaged  during 
(a)  the  previous  two  weeks  and  (b)  the  rest  of  the  previous  year.  Attention 
was  directed  first  to  the  previous  two  weeks  to  obtain  information  that  was 
as  specific  as  possible  for  a  period  of  time  oi  quite  recent  memory  and  to 
establish  a  pattern  in  which  the  interviewee  dealt  with  information  as 
specific  as  possible. 

The  list  of  major  areas  of  activity  provided  the  framework  within 
which  the  interviewer  could  proceed  to  obtain  detailed  information  about 
tasks  and  competencies.   After  obtaining  this  listing,  the  interviewer 
asked  for  a  description  of  each  of  the  tasks  which  were  part  of 
each  of  the  major  areas  of  activity.   Along  with  the  description  of  each 
task,  the  interviewee  was  asked  to  identify  each  of  the  competencies  he 
employed  in  accomplishing  that  task.   Information  about  tasks  and  competencies 
was  recorded  on  copies  of  the  interview  data  sheets,  with  sufficient  copies 
being  used  to  record  the  tasks  and  competencies  which  were  a  part  of  all  of 
the  major  areas  of  activity  previously  identified. 
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The  "instructions  to  interviewers"  given  below  provides  a  fuller 
picture  of  the  specified  procedures  followed  by  the  interviewers. 

During  the  brief  introductory  discussion  in  which  you  develop  rapport 
with  the  interviewee,  explain  that  the  interview  is  part  of  an  extensive 
effort  to  ascertain  the  nature  of  the  activities  in  which  educational  research 
and  research-related  personnel  are  engaged.    Explain  that  the  organization 
in  which  he  is  employed  was  selected  as  an  exemplary  center  of  research, 
development,  diffusion  and/or  evaluation  work  and  we  would  like  to  spend  some 
time  with  him  to  ascertain,  as  clearly  as  possible,  the  tasks  iii  which  he  is 
engaged  and  the  competencies  required  to  accomplish  these  tasks.  Explain 
the  need  to  tape  record  the  interview  and  arrange  the  tape  recording  equipment. 

After  recording  the  preliminary  information  required  on  the  first  half 
of  the  interview  record  form,  ask  the  interviewee  to  list  the  major  projects 
or  areas  of  activity  in  which  he  has  been  engaged  during  the  past  two  weeks. 
If  he  has  a  daily  calendar  of  his  activities  available,  it  may  serve  as  a 
reminder  of  activities  he  otherwise  might  forget.    These  activities  should 
be  listed  in  the  space  provided  on  the  bottom  half  of  the  interview  record 
form.   A  name  or  phrase  describing  the  activity  is  all  that  is  needed  at 
this  point  since  this  information  will  be  used  only  as  a  means  of  facilitating 
the  remainder  of  the  interview. 

The  next  step  is  to  begin  recording  on  interview  data  sheets  the  tasks 
and  competencies  which  are  related  to  the  major  activities  mentioned  above. 
Beginning  with  the  first  activity  mentioned  earlier,  ask  the  interviewee  to 
describe  the  various  tasks  in  which  he  has  been  engaged  as  part  of  the  activity. 
On  the  interview  data  sheets  list  these  tasks  and  ask  the  interviewee  to 
describe  the  competencies  (skills  or  knowledge)  required  to  perform  each  of 
these  tasks. 

It  is  not  expected  that  the  information  Solicited  within  the  scope 
of  a  given  activity  will  be  acquired  in  any  particular  order.  Information 
on  several  tasks  may  be  acquired  before  specific  information  is  acquired 
about  the  related  competencies,  or  full  information  about  the  competencies 
for  each  task  may  be  acquired  before  going  to  the  next  one.    Full  information 
within  each  major  activity  must  be  acquired,  however,  before  proceeding  to 
the  next  activity. 

After  you  have  reviewed  the  interview  data  sheets  for  a  given  activity 
to  be  sure  you  have  acquired  the  information  needed  for  that  activity,  pro- 
ceed to  the  next  activity  listed  on  the  interview  record  form.   In  the  case 
of  the  second  and  subsequent  activities,  the  procedure  followed  should  be 
the  same  as  the  first  with  one  exception  —  the  only  tasks  that  need  be 
investigated  are  those  which  have  not  been  mentioned  in  earlier  activities. 
Thus,  a  task  should  be  recorded  on  the  interview  data  sheets  only  once  even 
though  mentioned  in  connection  with  more  than  one  activity. 
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After  the  above  process  has  been  completed,  the  interviewee  should 
be  led  through  the  same  process  for  other  areas  of  activity  and/or  tasks 
in  which  he  has  been  engaged  during  the  past  year  but  which  have  not  been 
contained  within  the  two-week  period.   This  portion  of  the  interview 
generally  takes  less  time  than  the  former  part.    The  use  of  this  two-part 
procedure  focuses  attention  on  the  period  of  time  which  is  more  likely 
to  be  clearly  remembered  but  also  collects  information  about  a  period  of 
time  that  may  not  be  representative  of  the  normal  activities  of  the  inter- 
viewee. 

Since  the  focus  of  this  study  is  research  and  research-related 
activities,  functions  such  as  teaching,  administration,  and  consulting 
are  outside  the  purview  of  our  inquiry  unless  they  contribute  directly 
to  sonie  identifiable  aspect  of  educational  inquiry  or  inquiry-related 
activities  fier  se_.    For  example,  an  interviewee  may  serve  as  a  consultant 
to  a  researcRproject;  we  are  interested  in  the  tasks  that  he  may  perform 
in  that  role  that  are  a  part  of  the  research  effort.  The  teaching  of  a 
college  course  on  educational  research  methods,  however,  is  outside  the 
scope  of  our  concern  since  it  is  not  generally  a  direct  contribution  to  a 
specific  identifiable  research  study.    In  the  case  of  a  director  of  a 
research  project  we  are  interested  in  his  direct  research  administration 
contributions  but  not  in  general  administrative  functions  such  as  building 
budgets,  alloting  work  loads,  and  attempting  to  maintain  staff  morale  — 
duties  that  seem  representative  of  any  general  administrative  role. 
Another  area  of  importance  that  is  outside  the  scope  of  our  efforts  is 
the  affective  area,  including  such  components  as  interest  in  j^search, 
curiosity,  and  tolerance  of  ambiguity.    Although  important  in  research  and 
research -related  work,  they  are  outside  the  scope  of  this  investigation. 

You  must  use  care  to  avoid  suggesting  responses  to  the  interviewee. 
If  an  interviewee's  statements  are  unclear  to  you,  ask  for  greater  detail 
rather  than  suggesting  any  response  to  him.  ^ 

Training  of  Interviev^ers.    Six  interviewers  were  trained  to  conduct 
the  interviews  as  described  above.   A  set  of  statements  to  be  used  by  an 
interviewer  during  an  interview  and  based  upon  the  ideas  given  in  the  set 
of  directions  above  was  written  and  used  in  the  training  process.  Although 
the  interviewer  was  not  expected  to  memorize  the  statements  and  use  them 
verbatim,  their  use  in  practice  sessions  encouraged  uniformity  among  the 
interviewers.   The  training  consisted  of  the  following: 

1.      After  extensive  discussion  with  the  developers  of  the  interview 
procedure  concerning  its  goals  and  mode  of  operation,  each  trainee  listened 
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to  a  tape  recording  of  an  actual  interview.   Additional  discussion  followed 
and  the  process  of  listening  to  tapes  and  discussion  of  them  was  repeated 
until  the  trainee  was  judged  to  be  prepared  adequately  for  the  next  step 
in  the  training  process. 

2.  The  trainee  then  observed  as  one  of  the  developers  of  the  inter- 
view process  conducted  an  interview,  for  training  purposes,  with  a  member  of 
an  RDDE  agency.    Observation  of  the  interview  was  followed  by  discussion  of 
the  procedures  employed.    Again,  the  process  was  repeated  until  the  trainee 
was  prepared  to  conduct  a  trial  interview  himself. 

3.  Next,  the  trainee  conducted  a  trial  interview  while  being 
observed  by  one  of  the  developers  of  the  procedure.   The  critique  of  his 
interview  which  followed  provided  the  interviewer  with  a  basis  for  revising 
his  approach.    Repetition  of  this  step  continued  until  the  interviewer  was 
judged  to  be  fully  trained. 

Sources  of  Data 

In  the  view  of  the  Task  Force,  it  was  paramount  to  interview  persons 
who  were  doing  exemplary  work  in  educational  reseandi  and  research-related 
areas.    The  competencies  possessed  (and  used)  by  good  researchers,  developers, 
evaluators,  and  diffusers  were  of  primary  interest.^    Therefore,  two  strate- 
gies were  used  to  identify  persons  doing  exemplary  work  in  all  categories. 
First,  the  Task  Force  used  ratings  of  400  recent  research  articles  to  identify 
the  13  authors  who  were  rated  as  doing  the  best  research  studies.   These  13 
persons  were  found  to  be  widely  scattered  throughout  the  country  and  the 

^Interviewing  unproductive  researchers  as  a  comparison  group  may 
provide  useful  data  for  a  future  study,  particularly  when  the  affective 
characteristics  are  considered.    Such  comparisons,  however,  were  clearly 
beyond  the  scope  of  the  present  study. 
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interviews  were  conducted  by  telephone.*^    Second,  14  agencies  were  identified 
by  the  Task  Force  as  doing  exemplary  work  in  educational  RDDE.    Knowledge  of 
the  quality  of  work  generally  came  through  first-hand  experience  of  several 
Task  Force  members  with  each  site,  supplemented  in  some  cases  by  information 
acquired  through  the  1969-70  telephone  interviews  with  employers  and  super- 
visors in  research  and  related  institutions. 

Each  of  the  14  agencies  was  contacted  and  permission  requested  to 
conduct  personal  interviews  with  seven  or  eight  of  their  staff  sometime  dur- 
ing January  or  February  of  1971.    Cooperation  was  received  in  every  instance. 
The  final  list  of  agencies  visited  and  the  Task  Force's  a  priori  perception 
of  the  functional  emphases  within  each  agency  are  shown  in  Table  1.  Within 
each  of  the  organizations,  a  representative  group  of  professional  personnel 
was  selected  through  consultation  with  the  director  of  the  project.  The 
intent  was  to  obtain  a  cross-section  of  the  different  types  of  personnel 
(e.g.,  evaluators,  materials  developers,  and  diffusers)  employed  at  various 
levels  (e.g.,  directors,  professional  full-time  employees,  assistants) 
within  each  exemplary  setting.    A  total  group  of  116  persons 
was  selected  —  the  maximum  possible  under  existing  budgetary  constraints. 


The  InterviiBW  procedure  described  above  was  found  to  need  no 
modifications  when  interviews  were  conducted  by  telephone. 
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Table  1 


Organizations  Within  Which  Interviews  Were  Conducted 
And  Perceptions  of  Functional  Emphases  In  Each  Agency. 


American  Institutes  for 
Research,  Pittsburgh 

Learning  Research  and 
Development  Center, 
PI ttsburgh 

National  Curriculum  Pro- 
jects, Boulder,  Colo. 
(BSCS,  SSEC) 

Southwest  Regional  Labor- 
atory (SWRL),  Los  Angeles 

Research  for  Better  School^,, 
Philadelphia 

New  York  State  Department 
of  Education,  Albany 

Nassau  County  BOCES  Center, 
Jericho,  New  York 

Columbus  Public  Schools 
Research  &  Evaluation 
Dept. ,  Columbus,  Ohio 

Pittsburgh  Public  Schools 
Office  of  Research, 
Pittsburgh 

Center  for  Instructional 
Research  and  Curriculum 
Evaluation  (CIRCE), 
Urbana,  Illinois 

Educational  Policies  Research 
Center,  Syracuse,  New  York 

Vocational  &  Technical  Educ- 
ation ERIC  Center,  Columbus, 
Ohio 

University  of  Virginia 
Evaluation  Center, 
Charlottesville 

Montgomery  County  Public 
Schools  Office  of  Research, 
Rockvllle,  Maryland 

Exemplary  researchers  Identi- 
fied as  Individuals 


Research 
P 


Development   Diffusion  Evaluation 


NOTE:   P  =  primary  function  and  S  =  secondary  function 
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Compilation  and  Categorization  of  the  Data 
Processing  of  the  data  began  during  the  interviews  in  that  (1)  the 
Interviewer  made  a  written  record  of  the  interviewee's  responses  on  the 
interview  record  form  (see  Appendix  A)  and  interview  data  sheets  (see 

Appendix  B)  and  (2)  the  interviewer  made  some  interpretations  of  the  inter- 

} 

vieWee's  remarks,  i.e.,  he  did  not  necessarily  record  them  verbatim.  Within 
24  hours  after  the  interview,  the  interviewer  listened  to  the  tape  record- 
ing of  the  interview  to  check  the  accuracy  of  his  written  record.  Addi- 
tions, deletionis,  and  modifications  were  made  in  the  written  record  as 
necessary.    All  audio  tapes  and  interviewee  forms  were  retained  for  later 
compilation  and  analysis  work. 

Tasks.    Prior  to  any  analysis  of  the  data  acquired  in  the  interviews, 
a  system  of  classifying  and  compiling  the  data  had  to  be  developed  and 
employed.   The  categories  of  research,  development,  diffusion,  and  evalua- 
tion (RDDE)  are  coninonly  employed  to  describe  the  activities  of  educational 
research  and  research-related  personnel  but  there  has  been  no  empirical 
verification  of  the  apparent  logic  and  appropriateness  of  these  categories 
for  describing  such  activities.    Rather  than  impose  the  RDDE  rubric  on  the 
data  collection  process  and  collecting  task  and  competency  data  within 
such  a  framework,  the  task  and  competency  data  were  collected  independently 
of  any  such  set  of  categories...  The  intent  then  was  to  analyze  task  data 
to  determine  what  groupings  of  tasks  (i.e.,  functions)  actually  exist. 
The  initial  step  in  this  analysis  was  the  logical  formulation  of  a  set  of 
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.69  categories  into  which  the  entire  collection  of  tasks  from'the  116 
interviews  could  be  classified.    This  set  of  categories  was  developed 
jointly  by  the  six  persons  who  previously  had  conducted  the  interviews. 
A  list  of  the  task  categories  and  a  description  of  each  are  presented  in 
Appendix  C. 

After  task  categories  were  formulated,  the  interviewers  coded  each 
of  the  tasks  identified  in  their  own  interviews  according  to  the  set  of  69 
task  categories.    For  purposes  of  training  in  categorizing  the  data,  twelve 
tape-recorded  interviews  were  listened  to  and  coded  by  the  six  interviewers 
(two  interviews  per  interviewer).    Then  one  tape,  encompassing  the  broadest 
range  and  greatest  number  of  tasks,  was  selected  for  conducting  a  reliabil- 
ity check.   The  six  interviewers  independently  (1)  listened  to  the  taped 
interview  and  recorded  the  data  on  forms,  and  (2)  coded  the  data  according 
to  the  69  task  categories.   Using  analysis  of  variance^,  the  average  inter- 
judge  reliability  was  computed  on  the  classified  tasks.    The  average 
inter-judge  reliability  was  found  to  be  .71.    The  results  of  this  analysis 
are  reported  in  Table  2. 


^Winer,  B.  J.    Statistical  principles  in  experimental  design. 
New  York :   McGraw-Hi 1 1,  1962,  pp.  124-128. 
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TABLE  2 


Analysis  of  Variance:   Inter-rater  Reliability  for  Tasks 


SOURCE 


df 


ms 


Between  Tasks 
Within  Tasks 
Between  Judges 
Residual 


345 


5 

340 


68 


.56 
.16 


.38 
.15 


.16 
.56 


.71 


Competencies.    The  compilation  and  categorization  of  the  competency 
data  were  conducted  in  a  manner  similar  to  that  described  above  for  the 
task  data  with  the  exception  that  the  coding  of  the  data  was  done  by  one 
person.   On  the  basis  of  the  data  from  thirty  interviews,  four  members  of 
the  project  staff  (two  interviewers,  the  project  director,  and  the  data 
analyst)  formulated  a  preliminary  set  of  categories  to  use  in  classifying 
the  competencies.   To  facilitate  the  categorization  of  competencies, 
twelve  general  categories  were  first  formed.    The  specific  competencies 
identified  from  the  thirty  interviews  were  then  listed  within  the 
appropriate  general  category  to  which  they  belonged.  The  competencies  for 
the  remaining  86  Interviews  then  were  categorized  by  the  coder,  this  person 
being  one  who  had  participated  in  the  formulation  of  the  preliminary  set 
of  categories  and  who,  as  an  interviewer,  had  conducted  the  greatest  number 
of  interviews.   The  classification  system  was  expanded  slightly  as  the 
classification  proceeded  to  allow  for  the  appropriate  inclusion  of  a  few 
additional  competencies  that  had  not  occurred  in  the  first  thirty  sets  of 
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data  from  which  the  classification  system  was  developed.  The  final  set  of 
226  competency  categories  is  presented  in  Appendix  D. 

A  reliability  check  on  the  competency  data  was  conducted  in  the 
same  manner  as  that  described  above  for  the  task  data.   One  tape  was 
selected  as  having  the  broadest  range  and  greatest  number  of  competencies. 
Competency  data  from  this  tape  were  recorded  on  the  interview  data  sheets 
by  the  six  interviewers.    They  then  classified  the  data  according  to  an 
initial  form  of  the  competency  classification  system  and  the  six  sets  of 
categorized  data  were  checked  for  reliability  by  the  same  procedure  des- 
cribed earlier,  resulting  in  an  inter-judge  reliability  of  .68.  Results 
of  this  analysis  are  reported  in  Table  3.     Although  only  one  interviewer 
coded  the  data,  the  reliability  check  indicated  that  recording  interview 
data  on  the  forms  was  adequately  reliable. 


TABLE  3 

Analysis  of  Variance: 
Inter-rater  Reliability  for  Competencies 


SOURCE 

df 

ms 

Between  Competencies 

'109 

.56 

Within  Competencies 

550 

.18 

Between  Judges 

5 

.55 

Residual 

545 

.17 
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Each  Interviewee  was  scored  on  each  of  the  69  task  categories 
Identified  In  Appendix  C.    If  the  Interviewee  Indicated  that  he  had 
performed  or  was  performing  a  given  task,  he  was  given  a  score  of  one. 
Otherwise  the  task  score  given  was  zero.    These  scores  were  assigned  by 
the  person  who  conducted  the  interview.    The  competency  data  scores  were 
assigned  In  the  same  way  as  for  tasks.    However,  for  the  competencies,  the 
scores  were  assigned  by  the  Interviewer  who  was  responsible  for  the  categor- 
ization of  the  original  competencies.   The  categorized  data  for  both  the 
tasks  and  competencies  were  transferred  to  computer  coding  sheets  by  seven 
specially  trained  clerks.    A  carefully  supervised  system  of  double  check- 
Ing  the  entire  scoring  procedure  was  employed  to  Insure  its  accuracy. 

Data  Analysis  Procedures 
The  analysis  of  the  deta  obtained  in  the  interviews  was  designed  to 
attain  the  objectives  described  earlier  in  this  chapter,  i.e.,  to  identify 
and  delineate  the  functions,  tasks,  and  competencies  (defined  above  under 
Definition  of  Terms)  which  are  required  of  research  and  research -related 
personnel . 

As  described  previously,  the  interviews  provided  data  on  the  tasks 
performed  and  competencies  used  by  research  and  research-related  personnel; 
no  attempt  was  made  to  logically  describe  the  functions  these  personnel 
performed.   The  Initial  facet  of  the  data  analysis  was  designed  to  find  what 
"clustering"  of  tasks  actually  occurred.    This  in  turn  would  define  functions 
performed.   The  analysis  Involved  a  factor  analysis  of  the  task  data. 

The  next  step  in  the  data  analysis  was  to  organize  the  226  categories 
of  competencies;  into  groupings  in  terms  of  thoir  common  possession  by 
research  and  resesirch-related  personnel  •    As  with  the  analysis  of  the  task 

44  ^ 
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data,  the  determination  of  groupings  of  competency  data  was  facilitated 
by  factor  analytic  techniques. 

The  final  major  step  in  the  data  analysis  was  to  relate  the  competency 
factors  to  the  task  factors  to  determine  what  competencies  actually  are  used 
in  performing  each  of  the  task  factors  (functions)  identified  through  the 
initial  factor  analysis.   These  three  stages  of  the  data  analysis  are 
described  in  greater  detail  below. 
Factor  Analysis  of  Task  Data 

The  69  tasks  listed  in  Appendix  C  were  factor  analyzed  to  ascertain 
possible  groupings  and  in  this  way  acquire  a  more  precise  description  of 
the  functions  —  collections  of  tasks  --  performed  by  research  and  research- 
related  personnel.   The  input  variables  consisted  of  69  dichotomiously  scored 
variables  on  116  subjects.    Factor  extraction  was  accomplished  by  a  principal 
axis  procedure;  squared  multiple  correlations  were  used  as  estimates  of  the 
communal i ties.     An  oblique  transformatipn  was  performed  using  the  Harris- 
Kaiser  Independent  Clusters  solution.    The  factors  to  be  transformed  were 
selected  using  the  Scree  Test.^    In  addition  to  performing  the  factor 
analysis,  simple  frequencies  for  each  task  were  obtained.^ 

The  matrix  of  phi  correlation  coefficients  among  the  69  tasks  is 
presented  in  Table  El  of  Appendix  E.   The  principal -axis  factor  extraction 
produced  48  factors  (corresponding  to  the  positive  eigenvalues  of  the 
intercorrelation  matrix  of  variables  with  squared  multiple  correlation 


The  Biomedical  computer  program  BMDOjJM,  General  Factor  Analysis, 
was  used  for  this  computation.    Dixon,  W.  J.  (EdTT"  BMP  biomedical  computer 
programs.   Berkeley  and  Los  Angeles:   University  of  California  Press,  fas. 
5 

Cattell,  R.  B.  (Ed.)    Handbook  of  multivariate  experimental 
psychology.    Chicago:    Rand  McNally  and  Company,  Id66,  p.  241.  

^The  Biomedical  computer  program  BMD04D,  Alphanumeric  Frequency 
JSUIJi,  was  used.  Dixon  W.  J.  (Ed.)  BMD  biomedical  computer  programs. 
Berkeley  and  Los  Angeles:   University  of  California  Press,  1968.  
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coefficients  along  the  diagonal)  which  accounted  for  73.8  percent  of  the 
total  variance.   The  results  Qf  the  Scree  Test  (presented  in  Figure  Fl  of 
Appendix  F)  revealed  that  12  factors,  accounting  for  45.4  percent  of  the 
total  variance,  should  be  retained  for  factor  transformation.    The  results 
of  the  oblique  transformation  are  presented  in  the  next  major  section  of 
this  chapter,  which  includes  the  results  of  the  data  analysis. 

Factor  Analysis  of  Competency  Data 

The  226  competencies  identified  in  the  interviews  with  the  116 
subjects  were  factor  analyzed  to  determine  possible  competency  groupings. 
Through  matching  these  competency  clusters  with  the  task  factors  identified 
above  it  was  hoped  that  it  would  b?;  possible  to  identify  the  competencies 
(skills  and  knowledge)  necessary  in  the  performance  of  each  general  task 
factor.    The  analysis^of  the  competencies  was  conducted  across  the  general 
task  factors  obtained  from  the  factor. analyses  of  the  tasks.    Due  to 
limitations  in  computer  capabilities  it  was  necessary  to  use  a  combination 
of  both  logical  and  empirical  procedures. 

The  first  step  in  this  process  involved  the  formation  of  three  groups 
of  competencies^  from  the  total  collection  of  226.    The  intent  was  to  group 
together  in  the  same  group  those  competencies  that  logically  would  be 
expected  to  be  associated  with  each  other.    The  three  groupings  were 
determined  by  the  four  members  of  the  project  staff  responsible  for  the 
categorization  of  the  original  competencies.   Several  of  the  competencies 
were  placed  in  more  than  one  grouping.    This  resulted  from  the  fact  that 
the  four  members  of  the  project  staff  felt  that  some  of  the  competencies 
logically  fit  with  more  than  one  group.    The  three  groups  of  competencies, 

oe      /j^"ce  the  factor  analysis  program  could  accommodate  a  maximum  of 
85  variables,  an  upper  limit  of  85  competencies  per  package  was  necessary. 
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identified  as  packages  01,  02  and  03  are  presented  in  Tables  E2,  E3  and 
E4  of  Appendix  E.® 

The  first  package  contains  64  competencies  which  for  the  most  part 
are  closely  related  to  data  analysis  and  computer  operations.   The  second 
package  contains  80  competencies  which  relate  primanly  to  evaluation, 
conceptual  and  management-administrative  competencies.   The  third  package 
contains  85  competencies  related  to  the  development  of  curriculum  materials, 
conceptual  activity,  inservice  education  and  management-administration. 
Several  competencies  thought  to  be  related  to  writing  skills  were  included 
In  both  the  second  and  third  packages. 

Each  of  the  three  packages  was  factor  analyzed  using  the  same 
procedure  as  previously  reported  for  the  task  data  analyses.   The  results  ^ 
of  the  factor  analyses  are  reported  in  Appendix  E.   Tables  E5,  E6  and  E7 
contain  the  matrices  of  phi  correlation  coefficients  for  the  three  compe- 
tency packages.   The  principal  axis  extraction  (smc's  along  the  diagonal) 
was  used  for  packages  01,  02  and  03.   Application  of  the  Scree  Test  (pre- 
sented in  Figures  F2,  F3  and  F4  in  Appendix  F)  revealed  that  8,  6  and  6 
factors  (accounting  respectively  for  42.4,  27.3  and  27.0  percent  of  the 
total  variance)  should  be  retained  for  factor  transformation.   Tables  E8, 
E9  and  ElO  contain  the  respective  factor  pattern  matrices,  and  Tables  Ell, 
El 2  and  El 3  contain  the  correlations  among  the  oblique  competency  factors 
for  the  three  packages. 

To  permit  the  competencies  in  the  three  packages  to  be  factor  analyzed 

In  one  comnon  analysis  (within  the  computer  constraints  of  85  variables),  the 

Slhlrty-eight  competencies  for  which  the  frequency  of  occurrence  was 
less  than  6  i.e.,  fewer  than  6  of  the  116  interviewees  listed  the  compe- 
tency —  were  not  included  in  this  analysis. 
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results  of  the  initial  three  factor  analyses  were  used  to  form  a  fourth 
package  of  competencies.   The  competencies  included  in  the  fourth  package 
were  selected  according  to  the  following  criteria: 

1.  Competencies  which  had  pattern  coefficients  above  .35  in  absolute 
value  and  which  were  in  at  least  two  of  the  three  initial  competency  factor 
patterns  were  included. 

2.  Competencies  which  had  pattern  coefficients  above  .48  in  absolute 
value  and  which  were  in  one  of  the^^hree  initial  factor  patterns  were  selected. 

3.  Competencies  were  selected  so  that  each  of  the  factors  in  each  of 
the  three  factor  patterns  was  represented  by  at  least  two  of  the  competencies 
which  loaded  on  it. 

Eighty-two  competencies  met  the  criteria  above  and  were  selected  for 
inclusion  in  package  04;  these  competencies  are  listed  in  Table  E14  in  Appendix  e. 

A  factor  analysis  of  package  04  competencies  was  accomplished  using  the 
same  procedures  as  reported  previously  for  the  analysis  of  the  task  data.  The 
matrix  of  phi  correlation  coefficients  among  the  82  competencies  appears  in 
Table  E15  in  Appendix  E.   The  principal -axis  factor  extraction  produced  59 
factors  (corresponding  to  the  positive  eigenvalues  of  the  intercorrelation 
matrix  of  variables  with  squared  multiple  correlation  coefficients  along  the 
diagonal)  which  accounted  for  80.2  percent  of  the  total  variance.   The  results 
of  the  Scree  Test  (presented  in  Figure  F5  of  Appendix  F)  revealed  that  seven 
factors,  accounting  for  33.5  percent  of  the  total  variance,  should  be  retained 
for  factor  transformation.    Identification  and  interpretation  of  the  trans- 
formed competency  factors  is  given  in  detail  in  the  next  major  section  of 
this  chapter. 

Jhe  processes  described  above  resulted  in  a  reduction  from  226 
competencies  to  82  competencies.    (Of  the  original  226,  38  were  excluded 
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because  of  low  frequencies  and  106  failed  to  meet  criteria  for  forming 
package  04  above.)    Since  these  reductions  were  largely  caused  by  limits 
in  computer  capacity  (e.g.,  necessity  of  holding  package  04  to  fewer  than 
85  competencies),  procedures  were  employed  to  use  the  results  of  the 
statistical  analysis  (e.g.,  correlation  matrices),  supported  where 
necessary  by  logical  analyses,  to  place  the  106  high-frequency  competencies 
not  previously  included  in  package  04  within  the  factor  pattern  obtained 
for  the  82  competencies  in  that  package.   The  following  three  criteria 
were  used  in  this  process: 

1 .  Factor  analysis  results  from  packages  01,  02  and  03 .  A 
competency  not  included  in  package  04,  but  present  in  package  01, 

02  and/or  03,  was  placed  with  those  competencies  with  which  it  loaded  in 
the  first  three  packages.   For  example,  competency  C301  was  not  included 
in  package  04  but  was  included  in  package  01.    In  package  01  it  loaded  on 
the  same  factor  and  in  the  same  direction  as  competencies  C801 ,  C807,  C815 
and  C826.   These  latter  four  competencies  were  included  in  package  04, 
loading  together  on  general  competency  factor  1.   Therefore,  C301  was 
placed,  albeit  post  hoc,  on  factor  1. 

2.  First-order  correlations.    Some  of  the  competencies  not  included 
in  package  04,  but  included  in  packages  01,  02  or  03,  did  not  possess  large 
pattern  coefficients  (greater  than  .35  in  absolute  value)  within  the  factor 
patterns  for  packages  01,  02  and  03.    However,  examination  of  their  first- 
order  correlations  revealed  that  many  of  these  competencies  correlated 
meaningfully  with  other  competencies  included  in  package  04  and  cut  across 
several  of  the  factors  in  the  appropriate  factor  pattern  in  package  01,  02 
and  03.    Ii  these  cases,  the  competency  was  included  in  more  than  one 
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factor  of  package  04.    For  example,  C212,  originally  included  in  package  01, 
was  placed  in  general  competency  factors  1  and  2  of  the  factor  pattern  for 
package  04  because  of  its  correlations  with,  other  competencies  within  these 
two  factors. 

3.   Logical  analysis.    Thirteen  of  the  original  226  competencies 
which  met  neither  of  the  above  criteria  were  placed  on  one  of  the  seven 
factors  using  logical  analysis.   The  four  persons  responsible  for  the 
analyses,  working  independently,  placed  each  of  these  competencies  on  one 
or  more  of  the  seven  general  competency  factors  which  resulted  from 
package  04.    Any  disagreements  were  resolved  at  a  meeting  of  the  four 
persons,  and  final  placement  of  these  competencies  reflected  unanimous 
agreement  of  the  four  judges. 

Results  of  this  analysis  are  reported  in  the  next  major  section  of 
this  chapter. 

Relating  General  Competency  Factors  to  General  Task  Factors 

The  third  and  final  stage  of  the  data  analysis  was  the  placement  of 
general  competency  factors  with  the  appropriate  general  task  factors  in 
order  to  identify  those  competencies  necessary  in  the  performance  of  each 
general  task  factor.    This  fit  between  task  and  competency  factors  was 
done  on  the  basis  of  (1)  the  frequency  of  occurrence  of  the  competencies 
in  the  performance  of  each  task  factor  and  (2)  the  results  of  the  factor 
analyses.   A  frequency  count  of  the  occurrence  of  each  competency  in  the 
performance  of  each  task  factor  was  performed.   This  provided  a  matrix 
(226  competencies  by  12  task  factors)  in  which  one  tally  was  entered  for 
each  person  who  employed  a  given  competency  in  the  performance  of  a  given 
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task  factor.  From  these  data  the  main  relationships  between  competency 

factors  and  task  factors  were  ascertained  by  the  two-stage  procedure 

described  below. 

Stage  one.   For  each  task  factor,  the  most  frequently  occurring 

q 

competencies  were  Identified.     Of  these  "high-frequency"  competencies, 
those  which  were  contained  In  one  of  the  competency  factors  (from  the 
package  04  factor  analysis)  were  then  Identified  and  the  number  of 
different  high-frequency  competencies  present  from  each  competency 
factor  was  Identified.   Those' competency  factors  having  the  largest 
number  of  high-frequency  competencies  were  judged  to  be  the  competency 
factors  most  frequently  employed  In  the  performance  of  the  given  task 
factor. 

The  number  of  competency  factors  Identified  with  each  task  factor 
varied  from  one  to  three,  the  criteria  for  Inclusion  being  the  presence 
of  three^^or  more  of  the  competencies  from  the  given  competency  factor  In 
the  group  of  high-frequency  competencies  under  the  task  factor. 

Stage  two.   Because  the  above  process  tended  to  favor  those 
competency  factors  which  contain  a  large  number  of  high-frequency  compe- 
tencies at  the  expense  of  competencies  which  occur  less  frequently  but 


.    ^This  number  ranged  across  categories,  from  10  competencies  to  25 
competencies;    in  all  but  one  case,  the  number  of  competencies  was  between 
18  and  25.    In  the  case  of  one  task  factor,  for  which  the  competency  fre- 
quencies were  very  low,  their  number  was  reduced  to  10  to  avoid  inclusion 
of  a  large  number  of  competencies  with  a  frequency  of  one. 

^^In  the  case  of  two  task  factors  for  which  the  total  number  of 
competencies  (from  package  04  competency  factors)  was  seven  or  less,  this 
number  was  reduced  to  two. 
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my  be  concentrated  on  one  task  factor,  a  supplementary  process  was  em- 
ployed to  identify  any  additional  competencies  that  ma^y  have  been  omitted 
due  to  this  bias.    Twenty-six  competencies  from  the  seven  competency  fac- 
tors were  Identified  which  had  not  occurred  frequently  enough  under  any 
one  task  factor  to  be  included  in  the  process  identified  above.    For  each 
of  these  competencies,  an  inspection  was  made  to  identify  those  competen- 
cies for  which  a  large  proportion  of  their  occurrence  was  concentrated  on 
one  task  factor.   The  specific  criterion  for  inclusion  was  that  the  fre- 
quency of  occurrence  on  a  given  task  factor  be  at  least  twice  as  great  as 
its  next  highest  frequency  of  occurrence  on  any  other  task  factor. (In 
all  cases,  this  frequenqy  of  occurrence  also  was  greater  than  one-fourth 
of  the  total  frequency  of  occurrence  across  all  twelve  task  factors.)  Final 
ly,  those  competency  factors  which  had  two  or  more  competencies  identified 
with  the  same  task  factor  were  judged  to  be  related  to  that  task  factor.  Thi 
process  resulted  in  the  addition  of  two  more  task  factor- competency  factor 
pairings  to  the  list  of  eighteen  which  resulted  from  the  earlier  process. 

The  results  of  this  two-stage  process  of  pairing  task  factors  and 
competency  factors  are  contained  in  the  next  section. 

Results  of  the  Data  Analysis 
This  section  is  sub-divided  into  three  parts  which  contain:  (1)  the 
results  of  the  factor  analysis  of  task  data,  (2)  the  results  of  the  factor 
analysis  of  competency  data,  and  (3)  discussion  of  attempts  to  relate  the 
results  of  the  task  and  competency  data  analyses. 

^^One  exception  was  a  competency  that  had  a  frequency  of  occurrence 
of  six  on  each  of  two  task  factors  and  a  frequency  of  one  on  only  one  other 
task  factor.    As  a  result,  it  was  identified  with  both  task  factors. 
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Identification  and  Interpretation  of  General  Task  Factors 

The  Harris-Kaiser  Independent  Clusters  transformation  of  the  twelve 
general  task  factors  resulted  in  the  oblique  factor  pattern  sunmarized  in 
Table  4.     The  twelve  factors  from  this  solution  are  identified  below 
together  with  a  listing  and  discussion  of  the  variables  which  loaded  on 
each  factor.    For  the  purpose  of  factor  identification  and  interpretation, 
only  tasks  which  had  factor  coefficients  greater  than  .33  in  absolute  value 
are  included. 

Task  factor  1.   The  principal  loadings  on  factor  1  are  from  the 
following  task  variables^ ^: 

T49   Using  computer  facilities  and  services  .716 

T46   Planning  and/or  selecting  data  analysis 

techniques  .559 

T  3   Conceptualizing  or  formulating  a  problem  or 

hypothesis  for  empirical  studies  .472 

T51    Interpreting,  reviewing,  and  integrating 

the  results  of  data  analysis  .455 

T  6   Formulating  a  design  for  a  research  study  .414 

T47   Conducting  data  analyses  by  non-computerized 

methods  .391 

T38   Conducting  interviews  -.379 

T43   Reviewing  and  critiquing  extant  educational 

programs  and  products  -.375 

T48   Developing  a  computerized  data  bank  and 

retrieval  system  .374 

Each  task  which  loads  positively  on  this  factor  is  related  to  empirical 

research,  with  the  heaviest  emphasis  being  upon  design  and  data  analysis. 


^%ask  numbers,  labels  for  each  task,  and  loadings  are  shown  under 
each  task  factor. 
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TABLE  4 

Factor  Pattern  for. Factor  Analysis  of  Tasks 
(Harris- Kaiser  Independent  Clusters  Oblique  Solution) 


TASK 

<  1 
1 

O 
C 

o 

A 

- 

b 

6 

FACTOR 
7 

8 

9 

10 

11 

12 

Tl 

191 

-137 

522 

-120 

135 

T2 

472 

209 

-179 

639 

147 

T3 

-142 

-203 

127 

-257 

271 

-156 

T4 

-213 

-208 

-150 

-500 

158 

181 

-224 

-108 

T5 

-187 

321 

510 

-114 

-151 

126 

-101 

-189 

T6 

414 

-107 

157 

101 

-189 

387 

136 

T7 

195 

-143 

549 

-156 

-175 

-103 

T8 

448 

523 

160 

-108 

108 

T9 

- 

671 

159 

112 

-125 

TIO 

-157 

460 

139 

120 

-364 

104 

Til 

275 

-167 

-118 

-254 

-380 

-221 

-130 

T12 

154 

539 

141 

220 

163 

T13 

190 

370 

124 

124 

T14 

-134 

-277 

443 

-202 

-181 

-182 

241 

-225 

-130 

153 

T15 

-136 

574 

183 

122 

121  . 

136 

T16 

263 

-191 

-164 

-135 

-252 

-157 

-197 

-135 

T17 

150 

335 

234 

110 

495 

T18 

754 

-127 

T19 

826 

T20 

-187 

346 

160 

232 

149 

-249 

152 

180 

T21 

190 

127 

584 

140 

-175 

-118 

T22 

102 

177 

626 

T23 

-138 

222 

289 

-180 

171 

164 

115 

407 

T24 

-261 

288 

-116 

-102 

441 

T25 

162 

633 

125 

141 

-137 

T26 

814 

161 

125 

T27 

-227 

-209 

-140 

-117 

-188 

-143 

195 

-162 

114 

T28 

763 

181 

T29 

597 

-173 

325 

-168 

-143 

189 

T30 

378 

-151 

-113 

386 

T31 

-104 

237 

-179 

-144 

106 

-126 

-249 

460 

T32 

-193 

-244 

105 

178 

-155 

240 

378 

T33 

292 

110 

409 

-196 

-199 

107 

T34 

283 

296 

283 

119 

381 

T35 

129 

133 

572 

-140 

-130 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in  absolute  value 
are  shovim.   Leading  decimal  points  are  omitted. 
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TABLE  4  (cont'd) 

Factor  Pattern  for  Factor  Analysis  of  Tasks 
(Harris-Kaiser  Independent  Clusters  Oblicjue  Solution) 


TASK 

1 
1 

n 
C 

3 

4 

S 

6 

FACTOR 
7 

8 

9 

10 

11 

12 

T36 

723 

-122 

-221 

145 

T37 

T38 

-379 

-106 

366 

-279 

-132 

124 

324 

T39 

-122 

429 

-312 

173 

279 

T40 

270 

-215 

157 

510 

174 

T41 

167 

-290 

320 

-190 

109 

401 

T42 

152 

-341 

123 

-373 

152 

-191 

-247 

T43 

-376 

223 

-208 

237 

-219 

-115 

183 

220 

243 

-116 

T44 

-168 

463 

-251 

-271 

111 

T45 

109 

-104 

-116 

523 

T46 

669 

220 

162 

111 

T47 

391 

-158 

-106 

-182 

142 

205 

T48 

374 

-124 

294 

-222 

T49 

716 

-112 

104 

171 

T50 

-155 

567 

T51 

455 

-133 

137 

-447 

T52 

157 

-104 

-125 

-102 

654 

T53 

319 

-287 

-114 

142 

-305 

-154 

TCil 

154 

-174 

151 

382 

-266 

316 

-231 

-101 

TCC 

-254 

-217 

227 

118 

277 

-177 

119 

-166 

1 3D 

194 

-161 

507 

-143 

T57 

-135 

343 

192 

-2G8 

192 

T58 

-142 

-171 

115 

167 

538 

254 

-150 

T59 

134 

205 

177 

224 

289 

105 

202 

T60 

202 

-255 

111 

203 

-335 

147 

218 

T61 

114 

158 

272 

276 

151 

-240 

196 

-169 

T62 

464 

-195 

-177 

-215 

-118 

T63 

-220 

236 

-236 

349 

138 

•125 

-189 

-281 

-108 

T64 

-123 

-205 

361 

-203 

-291 

-181 

293 

T65 

-106 

585 

122 

240 

T66 

t 

891 

T67 

811 

T68 

125 

-209 

109 

-131 

-191 

-277 

129 

368 

125 

T69 

317 

226 

-170 

259 

VARIANCE 

OF 

3.227  3.764  3.061  3.473  2.081  2.607  2.340  2.127  2.047  2.210  2.260  2.140 

FACTORS 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in  absolute  value 
are  shown.   Leading  decimal  points  are  omitted. 
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Tasks  49,  46,  47  and  48  all  deal  with  some  aspect  of  computer  utilization 
or  data  analysis.   Tasks  3  and  6  deal  with  formulation  of  a  problem  or 
research  design,  while  the  remaining  variable  with  a  positive  loading, 
task  51,  deals  with  interpretation  of  data  and  data  analysis.  Therefore, 
task  factor  1  is  named   designing  research  studies  and  conducting  and 
interpreting  data  analyses. 

Task  factor  2.   Task  factor  2  is  named  developing  instructional 
materials   in  accord  with  the  following  variables  which  possess  high 
loadings  on  it; 

T26  Specifying  and  sequencing  learning  activities  .814 

T28  Designing  curriculum  materials  .753 

T25   Identifying  and  formulating  educational 

objectives  .633 

T29  Writing  or  revising  printed  curriculum 

materials  597 

T44   Field  testing  of  curriculum  materials  .463 

T  8   Designing  a  specific  educational  development 

activity  443 

T30  Developing  non-textual  learning  materials  .373  ' 

Each  of  these  variables  is  clearly  part  of  the  process  of  developing 
instructional  materials. 

Task  factor  3.   The  following  variables  loaded  on  this  factor; 

T21    Supervising  professional  personnel  .584 

T15  Allocating  human  and  material  resources  to 

activities  574 

T12  Formulating  budgets  and  conducting  cost- 
analyses  .539 

T  8   Designing  a  specific  educational  development 

activity  .523 
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T  5 

Identifyina  a  oroduct  or  Dpoaram  which  nppHc 

•  ^ "WBi^  "'"^    ^               www   wi     |i/ 1  wH  1  Will  ft  1 1  1  VI 1    ilCBU w 

to  be  developed 

.510 

T14 

Engaging  in  executive  planning  and  policy- 
making 

.443 

T13 

Utilizing  a  PERT  or  other  management  planning 
system 

.370 

T20 

Orienting,  training  and  upgrading  project 
personnel 

.346 

T42 

Tabulating  and  categorizing  data 

-.341 

the  eight  variables  which  load  positively  on  the  factor,  including 

the  three  highest  loadings,  refer  to  management  or  administration  tasks. 
The  other  two  with  positive  loadings  (T8  and  T5)  are  development  tasks 
with  an  emphasis  on  those  aspects  of  development  in  which  a  higher  level 
administrator  likely  would  be  involved.    Severalj  of  the  positive  loadings 
are  on  variables  which  have  to  do  with  broad-scale  planning,  policy- 
making and  supervision.   To  distinguish  this  factor  from  task  factor  11 
(which  is  also  an  administration  and  management  factor  but  focused  on 
tasks  that  are  characteristic  of  persons  who  have  a  lower  level  position 
in  a  project  or  program  hierarchy),,  general  task  factor  3  is  named  "first- 
level"  administration  of  inquiry  and  inquiry-related  projects  and  activities. 

Task  factor  4.    The  highest  loadings  on  this  factor  are  from  the 
following  variables: 

T36  Selecting  or  constructing  and  revising 
questionnaires,  checklists,  interview 
schedules,  and  observation  systems  .723 

T  9  Conducting,  managing,  or  monitoring  a 

formative  evaluation  .671 

T35  Selecting  or  constructing  and  revising 

measures  of  affect  .572 

T  7   Formulating  a  design  or  plan  for  an  evaluation  .549 
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TIO  Conducting,  managing,  or  monitoring  a  summatlve 

evaluation  .460 

T39  Administering  group  tests  and  collecting  data 

by  use  of  paper  and  pencil  Instruments  .429 

T33  Planning  data  collection  procedures  .409 

T38  Conducting  Interviews  .366 

T63  Conducting  Inservlce  education  programs  .349 

The  eight  variables  which  load  highest  on  this  factor  specifically  refer 

to  either  evaluation  tasks  or  data  collection  tasks  which  are,  of  course, 

a  necessary  part  of  evaluation  activities.   Although  many  of  the  types  of 

data  collection  specified  In  the  tasks  (e.g.,  T36,  T35,  T39  and  T38)  are 

common  techniques  for  collecting  research  data,  their  loading  with 

evaluation  tasks  on  this  factor  suggests  that  they  may  be  used  more 

frequently  by  evaluators  ~  Indeed,  some  of  these  techniques  may  be  among 

the  evaluator's  most  useful  tools.   Therefore,  task  factor  4  Is  named 

conducting  evaluations  and  constructing  and  using  data  collection 

Instruments. 

Task  factor  5.    The  highest  loadings  on  this  factor  are  from  the 
following  variables: 

T58  Disseminating  Information  about  activities 

on  a  specific  project  or  In  a  specific  agency  .538 

T  4  Conducting  philosophical  and  historical 

analyses  -.500 

T54  Preparing  and  delivering  a  lecture  or  oral 
presentation  on  a  research  or  research- 
related  topic  .382 

T42  Tabulating  and  categorizing  data  -.373 
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T64  Translating  written  materials  from  one  language 

to  another  .361 

T57   Preparing  reports,  educational  materials,  and 

other  printed  materials  for  mass  production  .343 

T17   Preparing.  RFPs   and  guidelin;?s  for  preparation 

of  proposals  .335 

The  largest  loadings  are  related  to  tasks  58  and  4.    There  appears  to  be 

no  compelling  reason  for  these  two  variables  to  load,  albeit  in  opposite 

directions,  on  the  same  factor. However,  in  deference  to  the  slightly 

higher  (in  absolute  value)  loading  of  T58  and  the  fact  that  the  four 

remaining  positive  loadings  are  on  variables  clearly  related  to  diffusion, 

task  factor  5  is  named  diffusing  information  and  products. 

Task  factor  6.    The  three  principal  loadings  on  this  factor  are  from 

the  following  variables: 

T66  Selecting  and  indexing  documents  for  inclusion 

in  information  storage  and  retrieval  systems  .891 

T67  Writing  abstracts  of  materials  selected  for 
inclusion  in  an  information  storage  and 
retrieval  system  .811 

T65   Developing  and  modifying  information  storage 

and  retrieval  systems  .585 

All  three  of  these  variables  are  tasks  that  are  part  of  developing  and 

operating  information  storage  and  retrieval  systems  such  as  ERIC  centers. 

Thus,  general  task  factor  6  is  named  developing  and  operating  information 

storage  and  retrieval  systems. 


^  It  could  be  that  task  4  (and  task  42)  refer  to  activities  more 
likely  to  be  pursued  by  someone  conducting  research  per  se,  an  unlikely 
pursuit  for  a  disseminator.    However,  this  must  be  vTeweTlis  speculation 
at  this  point. 
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Task  factor  7.   The  principal  loadings  on  this  factor  are  from  the 
following  variables: 

T19  Processing  proposals  for  funding  and  monitor- 
ing funded  proposals  .826 

T18  Reviewing  and  evaluating  proposals  submitted 

for  funding  .754 

T62   Conducting  on-site  evaluation  visits  .464 

The  first  two  variables  reflect  activities  engaged  In  by  persons  responsible 

for  dispensing  funds  to  support  Inquiry  and  inquiry-related  activities  in 

education.   The  analysis  suggests  that  these  same  persons  also  conduct  on-site 

evaluation  visits  to  projects  or  programs  which  they  have  funded.  Task  factor  7 

Is  named  evaluating  Inquiry  and  Inquiry-related  proposals  and  monitoring 

funded  projects. 

Task  factor  8.   The  variables  which  have  the  highest  loadings  on 
tills  factor  are  as  follows: 

T  2  Utilizing  formal  search  procedures  to  acquire 

Information  .639 

T  1    Reading  the  literature  and  acquiring  up-to- 
date  Information  through  other  means  .522 

T56  Reviewing  and  evaluating  research  and 

research-related  reports  .507 

The  first  two  variables  deal  with  the  acquisition  of  Information.    The  third 

variable  Is  the  task  of  Initially  reviewing  reports  and  making  judgments 

about  the  information  in  them.   Therefore,  general  task  factor  8  is  named 

searching,  reading,  and  reviewing  the  literature. 

Task  factor  9.   Task  factor  9  is  named  designing  and  maintaining 

computer  systems  and  writing  computer  programs.    The  Important  loadings 

on  this  factor  are  as  follows: 
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T52 


Designing  computer  programs  and  programning 


.654 


T50 


Di'tsigtilng  computer  systems,  inspecting  and 
diagnosing  computer  problems,  and  repairing 
computer  equipment 


.567 


Til 


Writing  a  proposal 


.380 


This  factor  shows  that  the  designing  of  both  computer  systems  and  programs, 
as  well  as  the  maintenance  of  systems  and  programming  Itself,  tend  to  be 
done  by  the  same  persons.  Such  persons  are  typically  specialists  not 
involved  in  writing  a  proposal  and,  therefore,  it  is  not  startling  that 
task  11  would  load  negatively  on  this  factor. 

Task  factor  10.   The  following  variables  possessed  high  loadings 
on  this  factor: 

T40  Administering. individual  tests  .510 

T17  Preparing  RFPs   and  guidelines  for  preparation 

of  proposals  .495 

T51    Interpreting,  reviewing,  and  Integrating 

the  results  of  data  analysis  -.447 

T  6   Formulating  a  design  for  a  research  study  .387 

TIO   Conducting,  managing,  or  monitoring  a 

suinnative  evaluation  -.364 

T60  Negotiating  with  publishers  and  equipment 

manufacturers  -.335 

It  is  not  readily  apparent  why  this  subset  of  task  variables  loaded  together 

on  this  factor.    Consequently  no  name  has  been  given  to  this  factor. 

Task  factor  11.   Task  factor  11  is  named  "second-level"  administration 

of  inquiry  and  inquiry-related  projects  and  activities.   The  principal 

variables  loading  on  this  factor  are  as  follows: 
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T22   Supervising  support  personnel 


.626 


T31    Producing  or  supervising  the  production  of 
curriculum  materials  In  quantity 


.460 


T24   Designing  and  selecting  facilities  and 
capital  equipment 


.441 


T23   Hiring  and  recruiting  personnel 


.407 


T41    Using  formal  or  Informal  observation 
systems  to  code  human  behavior 


.401 


T30   Developing  non-textual  learning  materials 


.386 


T32   Repairing  and  maintaining  equipment  (other 
than  computers) 


.378 


T68   Conducting  a  research  study 


.368 


The  first  four  variables  have  an  administrative  emphasis  as  Indicated  by 
their  Inclusion  of  supervision  (T22  and  T31),  selection  of  facilities  and 
equipment  (T24),  and  hiring  of  personnel  (T23).   As  noted  earlier,  this 
factor  seems  to  have  a  somewhat  different  character  than  task  factor  3, 
the  "first-level "administration  factor.  In  that  the  tasks  here  are  focused 
more  upon  tf!ie  management  of  on-going  activities  and  supervision  of  support 
personnel  (e.g. ,  clerks   and  technicians).   The  additional  four  variables 
that  load  on  this  factor  (albeit  with  lower  coefficients)  cover  a  variety 
of  operational  tasks  and  suggest  that  (1)  "second-level"  administrators 


are  found  In  a  variety  of  Inquiry  and  Inquiry-related  activities  and 
(2)  managers  at  this  level  often  are  directly  Involved  In  the  operations 
which  they  manage. 

Task  factor  12.   Task  factor  12  Is  named  developing  and  scoring 
tests  In  accord  with  the  following  variables  which  loaded  on  this  factor: 
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T37 


Scaling,  normlng  and  establishing  reliability 
and  validity  of  measuring  instruments 


.809 


T45 


Scoring  tests 


.523 


T34   Selecting  or  constructing  and  revising 

ability  and/or  achievement  tests  .381 

The  persons  engaged  in  this  group  of  tasks  appear  to  be  test  specialists 

who  are  principally  involved  in  the  development  of  tests. 

Intercorrelations  among  task  factors.    Study  of  the  intercorrelations 
among  the  twelve  factors  described  above  is  also  informative;    such  inter- 
correlations are  presented  in  Table  5.     Only  three  of  these  correlation 
coefficients  are  greater  than  .2  in  absolute  value;  these  three  are  all 
positive  correlations.   The  largest  of  these  correlations  (.270  between 
task  factors  1  and  4)  suggests  some  association  between  research  and 
evaluation,  which  is  not  surprising.    An  inspection  of  the  task  variables 
in  each  of  these  two  factors  points  to  the  use  of  data  as  the  strongest 
link  between  them.    The  next  strongest  of  these  correlations  (.229  between 
factors  3  and  11)  Is  between  the  two  administration  factors.    The  third 
strongest  association  (.208  between  factors  5  and  11)  suggests  that  some 
of  the  same  persons  who  are  engaged  in  "second-level"  administration 
activities  may  also  be  responsible  for  the  dissemination  of  information 
about  their  project  or  program. 

Task  factor  frequencies.   The  frequency  with  which  each  task  factor  . 
was  performed  by  the  subjects  in  the  sample  is  reported  in  Table  6.  Each 
subject  who  performed  one  or  more  of  the  tasks  loading  on  a  given  task 
factor  was  counted  once  and  only  orfte  in  the  tally  on  that  factor. 
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TABLES 

Number  of  Subjects  Who  Performed  One  or  More 
Tasks  Under  Each  Task  Factor 


IMoN  rAtlUK 

N  PERFORMING  TASK(S) 
UNDER  THIS  FACTOR 

1 

98 

2 

70 

3 

101 

4 

85 

5 

88 

6 

5 

7 

22 

8 

64 

9 

49 

10 

76 

11 

62 

12 

35 

In  view  of  the  fact  that  the  maximum  frequency  possible  in  Table  6 
for  any  task  factor  is  116  (the  number  of  subjects  in  the  study),  the 
overall  frequencies  appear  to  be  relatively  high.    In  addition  to  the  fact 
that  a  person  was  entered  in  the  tally  if  he  was  involved  in  even  one  task 
out  of  the  several  in  each  task  factor,  the  high  frequencies  suggest  that 
many  subjects  are  involved  in  a  variety  of  diverse  tasks.    This  point  is 
borne  out  by  inspection  of  the  original  data  and  is  also  consistent 
with  the  multiple  functions  of  many  of  the  agencies  in  which  members  of  the 
sample  were  employed. 
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The  two  factors  with  the  lowest  frequencies  and  the  two  factors 
with  the  highest  frequencies  deserve  specific  study  (although  not  simply 
because  of  the  extremity  of  their  occurrence).   The  low  frequency  (5) 
for  task  factor  6  —  developing  and  operating  Information  storage  and 
retrieval  systems  —  Is  apparently  due  to  the  fact  that  the  factor  Includes 
tasks  which  were  almost  exclusively  limited  to  members  of  one  ERIC  Infor- 
mation retrieval  center.    The  frequency  (22)  of  task  factor  7  —  evaluating 
proposals  and  monitoring  funded  projects  —  Is  higher  than  might  be 
anticipated  In  view  of  the  type   of  agencies  sampled.    However,  Inspection 
of  the  frequencies  of  each  specific  task  appearing  In  that  factor  (tasks 
18,  19  and  62)  Is  revealing.   Frequencies  for  each  specific  task  are  shown 
In  Table  7.     In  looking  at  task  19,  It  Is  apparent  that  only  seven  persons 
were  Involved  In  the  processing  of  proposals  and  monitoring  of  the  funded 
projects  ~  activities  peculiar  to  agencies  which  provide  some  funding 
support.    The  other  two  tasks  (T18  and  T62)  are  tasks  In  which  persons  In 
a  variety  of  agencies  are  often  engaged,  e.g.,  through  serving  on  review 
panels. 

At  least  two  factors  may  contribute  to  the  high  frequency  (101)  for 
task  factor  3  —  "first-level"  administration.    First,  this  task  factor 
Includes  a  relatively  large  number  of  tasks  (9).    Second,  It  may  Indicate 
that  many  persons  In  R  &^,D  organizations  enter  Into  "higher-level"  declslon- 
making  processes  at  least  occasionally  during  their  employment  In  the  type 
of  agencies  sampled.   Also,  the  sample  was  chosen  to  Include  a  number  of 
administrators  at  each  level,  resulting  In  many  persons  who  would  be 
routinely  expected  to  perform  higher-order  administrative  duties  as  part 
of  their  job.  The  relatively  high  frequency  (98)  for  task  factor  1  —  designing 
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TABLE  7 


Number  of  Subjects  Who  Performed 
Each  of  69  Specific  Tasks 


TASK        ^  PERFORM-  N  PERFORM- 

"^■^         ING  TASK  TASK  j^^^ 


Tl 

41 

T36 

47 

12 

44 

T37 

11 

T3 

40 

T38 

24 

14 

17 

T39 

24 

15 

31 

T40 

3 

16 

38 

T41 

17 

17 

47 

T42 

24 

T8 

28 

T43 

24 

19 

35 

T44 

24 

TIO 

19 

T45 

8 

Til 

39 

T46 

43 

T12 

28 

T47 

25 

T13 

31 

T48 

10 

T14 

49 

T49 

44 

T15 

55 

T50 

3 

T16 

25 

T51 

47 

T17 

5 

T52 

11 

T18 

11 

T53 

81 

T19 

7 

T54 

43 

T20 

30 

T55 

7 

T21 

42 

T56 

24 

T22 

26 

T57 

25 

T23 

28 

T58 

35 

T24 

8 

T59 

29 

T25 

37 

T60 

9 

T26 

18 

T61 

23 

T27 

12 

T62 

17 

T28 

24 

T63 

37 

T29 

22 

T64 

2 

T30 

17 

T65 

3 

T31 

8 

T66 

4 

T32 

3 

T67 

3 

T33 

43 

T68 

13 

T34 

31 

T69 

7 

T35 

18 

^  67 
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research  studies  and  conducting  and  Interpreting  data  analyses  —  Is 
probably  due  to  the  reliance  of  research,  evaluation  and  product  testing 
(part  of  development)  on  analyzing  data  and  Interpreting  results. 

Identification  and  Interpretation  of  Competency  Factors 

The  oblique  transformation  (Harris-Kaiser  Independent  Clusters 
solution)  of  the  seven  factors  obtained  from  the  factor  analysis  of 
competency  data  In  package  04  (as  described  In  the  previous  section) 
resulted  In  the  factor  pattern  summarized  In  Table  8.      The  factors  In 
this  pattern  are  Identified  below.    For  the  purpose  of  factor  Identifi- 
cation and  Interpretation,  only  competencies  which  possessed  pattern 
coefficients  greater  than  .30  In  absolute  value  were  used. 


I 
I 
I 


I 
I 
I 


ERIC 
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TABLE  8 

Factor  Pattern  for  Factor  Analysis  of  Package  04  Competencies 
(Harris -Kaiser  Independent  Clusters  Oblique  Solution) 


COMPETENCY 

1 

2 

3 

FACTOR 
4 

5 

6 

7 

CI  02 

325 

439 

210 

b  1  W 

-105 

1  WW 

C105 

416 

120 

160 

1  WW 

C106 

-118 

574 

-102 

1  W  mm 

371 

wr  1 

-100 

1  WW 

CI  07 

-131 

367 

-171 

160 

CI  09 

154 

449 

-167 

-114 

C115 

676 

C116 

467 

-121 

C202 

303 

-273 

102 

■  Wb 

262 

bWb 

144 

C203 

-182 

156 

-166 

234 

b  wT 

214 

b  1  ~ 

295 

b^W 

C205 

501 

138 

139 

207 

bW# 

175 

1  #  w 

-135 

■  WW 

C206 

332 

363 

-121 

1  b  1 

C208 

363 

263 

-141 

-281 

C209 

-156 

494 

-430 

~  WW 

135 

■  WW 

C211 

243 

206 

-108 

105 

223 

-166 

328 

C218 

166 

-269 

280 

-118 

133 

■  WW 

231 

bW  ■ 

C223 

345 

152 

159 

342 

Ww 

-122 

1  bb 

C302 

178 

249 

275 

-198 

278 

b  #  W 

-101 

•  w  • 

C304 

314 

197 

100 

1  WW 

230 

mm  WW 

-120 

■  bW 

151 

1  W  1 

C307 

261 

109 

166 

-257 

bW  W 

-178 

1  V  w 

456 

C310 

154 

-160 

471 

C312  X 

-140 

755 

C363 

661 

C371 

-146 

-256 

-121 

271 

245 

352 

C374 

633 

-188 

C404 

-151 

388 

155 

142 

-147 

C408 

413 

211 

111 

-195 

C409 

-159 

514 

113 

258 

C412 

520 

142 

166 

C413 

-111 

424 

C415 

-159 

158 

-208 

C417 

-105 

433 

-136 

-323 

177 

C420 

-148 

343 

-191 

285 

C501 

-209 

-159 

329 

225 

176 

C502 

-125 

-295 

345 

270 

293 

C541 

582 

193 

C543 

222 

584 

182 

-167 

C603 

> 

629 

r  ■ 

NOTE:  Only  factor  pattern  coefficients  greater  than  .100  In  absolute  value 
are  shown.   Leading  decimal  points  are  omitted. 
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TABLE  8  (cont'd) 

Factor  Pattern  for  Factor  Analysis  of  Package  04  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


COMPETENCY 


C604 

C607 

C608 

C609 

CIO 

C611 

C613 

C614 

C616 

C617 

C620 

C622 

£625 

C629 

C729 

C801 

C802 

C807 

C810 

can 

mu 

C815 
C819 
C822 
Cd24 
C825 
C826 
C828 
C830 
C831 
C832 


FACTOR 


1 

1 

2 

3 

4 

5 

6 

7 

-240 

243 

594 

-145 

-319 

325*- 

Wb  W 

-111 

487 

Twr 

373 

W/  W 

261 

bU  1 

-186 

1  OW 

-IRA 

-204 

304 

-105 

wV7 

lOft 
1  uo 

-232 

229 

-102 

547 

909 

bUb 

104 

113 

1  1  w 

1  OO 

-183 

-106 

110 

-100 

-112 

lib 

2?1 

H7 

9^1 

141 

112 

449 

-168 

333 

-243 

-210 

263 

-133 

-104 

-222 

356 

180 

249 

432 

-144 

238 

-248 

-236 

529 

179 

388 

286 

371 

282 

193 

-243 

-156 

283 

530 

138 

206 

-129 

145 

-176 

542 

571 

-112 

488 

136 

654 

-145 

210 

105 

565 

109 

-131 

-118 

650 

141 

-212 

130 

623 

-175 

-110 

-247 

466 

-158 

-285 

-137 

-103 

691 

-214 

-147 

164 

177 

149 

-182 

-315 

366 

558 

-151 

502 

-212 

-166 

-113 

292 

525 

-118 

-187 

NOTE: 


Only  factor  pattern  coefficients  greater  than  .100  in  absolute 
value  are  shown.    Leading  decimal  points  are  omitted. 
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TABLE  8  (con't) 

Factor  Pattern. for  Factor  Analysis  of  Package  04  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


FACTOR 


COMPETENCY 

1 

2 

3 

4 

5 

6 

7 

C911 

-103 

565 

-142 

C912 

-170 

462 

-128 

C916 

-217 

-188 

512 

C917 

328 

-145 

228 

-119 

200 

C001 

122 

-315 

478 

C002 

346 

-106 

179 

352 

C004 

511 

-202 

131 

115 

COOS 

292 

-250 

114 

194 

-139 

C009 

196 

392 

115 

-107 

C010 

281 

149 

-147 

444 

-148 

C013 

372 

461 

119 

155 

-111 

C015 

146 

388 

193 

107 

C017 

429 

-124 

193 

176 

-178 

C018 

290 

-281 

VARIANCE 
OF  FACTORS 

5.891 

4.033 

3.371 

3.537 

3.904 

3.133 

3.621 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  In  absolute 
value  are  shown.   Leading  decimal  points  are  omitted. 
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Competency  factor  1.    The  principal  loadings  on  this  factor  are  from 
the  following  competencies: 

C826   Ability  to  design  card  layouts  to  allow  data 
analysis  within  computer  constraints  and 
ability  to  use  standardized  computer  programs 
I  (e.g.,  BHD  series)  .691 

C824   Knowledge  of  how  computers  might  be  used  to 

analyze  data  .623 

C811    Knowledge  of  t- tests  and  critical  ratios  .571 

C822   Knowledge  of  alternate  methods  of  presenting 
statistical  data  (e.g.,  charts,  graphs,' or 
tables)  .565 

C830   Ability  to  use  computer  coding  .558 

C807   Knowledge  of  ANOVA  or  ANCOVA  designs  and 

techniques  .531 

C832   Ability  to  read  and  Interpret  computer  output  .525 

C831    Ability  to  keypunch  .502 


I 


i 


ERIC 


C205   Knowledge  of  questionnaire  construction 
techniques  and  appropriate  uses  for 
questionnaires  .501 

C815    Knowledge  of  factor  analysis  techniques  .488 

C825  Ability  to  allocate  time  and  money  wisely  In 

arranging  conq)uter  work  .466 

C017  Ability  to  describe,  explain,  or  elaborate 

in  writing  .429 

C408  Ability  to  formulate  a  rationale  to  support 

a  particular  position  or  argument  .413 

C801    Ability  to  choose  (or  design)  appropriate 

statistical  techniques  for  data  analysis  .388 

C013  Ability  to  write  in  an  interesting  or 

appealing  style  .372 
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C208   Knowledge  of  normlng  procedures  .363 

C002  Ability  to  revise  and  rewrite  .346 

C223  Ability  to  arrange  Items  In  a  format 

which  Is  easy  to  read  .345 

C206  Ability  to  construct  Instruments  to 
assess  attitudes  and  other  affective 
variables  .332 

C9T7   General  speaking  skills  .328 

CI 02   Ability  to  discuss  the  advantages  of 

establishing  evaluation  systems  In  educa- 
tional Institutions  .325 

C304   Knowledge  of  specific  experimental  and 

quasi -experimental  research  designs  .314 

The  competencies  which  possess  high  pattern  coefficients  on  this  factor 

Include  skills  and  knowledge  related  to  analysis  of  data  (C826,  C811,  C807, 

C815,  C801),  computer  operations  and  utilization  (C826,  C824,  C830/C831, 

C832.  C825),  Instrument  development  and  measurement  (C205,  C2()8,  C223, 

C206).  and  reporting  (C822»  C017,  C013,  C002).   Therefore,  factor  1  Is 

named  data  collection,  processing^  analysis  and  presentation  competencies. 

A  second  factor,  competency  factor  7,  also  has  Its  principal  loadings 

from  variables  similar  to  those  on  competency  factor  1.   Variables  with 

principal  loadings  on  factor  7  are  shown  below: 

C812   Knowledge  of  statistical  variance  and 

standard  deviation  .654 

C819  Knowledge  of  theoretical  assumptions  under- 
lying various  statistical  techniques  .650 

C810  IChowledge  of  statistical  regression  techniques  .542 

C307  Ability  to  design  studies  to  control 

extraneous  variables  .456 
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C801    Ability  to  choose  (or  design)  -pc^ropriate 

statistical  techniques  for  datu  analysis  .317 

C828   Knowledge  of  capablVfties  of  local  computer 

systems  .366 

C371    Ability  to  draw  or  compose  pictures  or 

Illustrations  for  curriculum  materials  .352 

can    Ability  to  construct  items  that  msasure 

what  one  sets  out  to  measure  .328 

The  competencies  which  possess  high  pattern  coefficients  on  this  factor 
include  skills  and  knowledge  related  to  analysis  of  data  (C812,  C819, 
C810,  C801),  computer  operation  and  utilization  (C828),  instrument  develop- 
ment (C211),  and  research  design  (C307).   The  highest  loadings  and  a 
majority  of  variables  relate  to  statistics  and  competency  factor  7  is 
therefore  named  statistical  competencies.   Admittedly,  there  is  a  large  area  of 
overlap  between  competency  factors  1  and  7  (r  =  .32).    The  difference  seems 
to  be  one  of  emphasis  rather  than  coverage,  with  loadings  on  factor  7 
clustering  primarily  on  only  one  of  the  areas  of  competency  Included  in 
factor  1. 

Competency  factor  2.   The  principal  loadings  on  this  factor  are  the 
following: 

CI  15   Ability  to  Incorporate  systematic  evaluation 
procedures  into  plans  for  developing  educa- 
tional programs  .676 

CI 06  Ability  to  work  with  public  school,  univer- 
sity or  state  department  of  education 
personnel  .574 

C613   Ability  to  identify  educational  needs  that 

should  be  addressed  by  educational  systems  .547 

CI  16  Ability  to  develop  techniques  for  providing 
evaluative  feedback  to  program  or  project 
personnel  in  time  to  allow  needed  modifica- 
tions to  be  made  during  the  operation  of 
the  program  .467 
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C109  Ability  to  help  others  Identify  and  state 

their  objectives  .449 

C102  Ability  to  discuss  the  advantages  of 

establishing  evaluation  systems  In  educa- 
tional Institutions  .439 

C629  Ability  to  vrark  effectively  with  decision 

makers  .432 

C105   Knowledge  of  personnel  and  the  organizational 
structures  of  public  school  systems  and 
universities  .416 

C009  Ability  to  put  quant1tat1ve,-or  numerical 

Information  Into  verbal  or  narrative  form  .392 

C107   Ability  to  determine  the  evaluative  questions 
which  must  be  asked  In  evaluation  and  the 
InformAlon  which  must  be  gathered  to  answer 
these  questions  .367 

C206  AMllty  to  construct  Instruments  to  assess 

attitudes  and  other  affective  variables  .363 

C622  Ability  to  facilitate  staff  work  on  an 

ongoing  project  .333 

COOl   Ability  to  write  -.315 

C610  Ability  to  Identify  and  obtain  resources 

needed  to  accomplish  program  objectives  .304 

C202  Knowledge  of  measurement  theory  and 

techniques  . 303 

The  competencies  which  possess  high  pattern  coefficients  Include  specific 

evaluation  competencies  (C613,  C116,  C109  and  CI 07),  competencies  relating 

to  Installing  evaluation  systems  (C115,  C102),  knowledge  of  some  tools  of 

evaluation  (C206,  C202),  competencies  In  relating  to  persons  necessary  In 

conducting  evaluations  (C106,  C629),  and  competencies  In  translating 

numerical  results  Into  reports  (C009).   Therefore,  competency  factor  2  is 

named  evaluation  competencies. 
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Conpetencv  factor  3.  The  highest  pattern  coefficients  on  this 
factor  are  from  the  following  variables: 


C911 

Ability  to  use  library  research  techniques 

.565 

C412 

Ability  to  predict  with  accuracy  the  Inpact 
of  an  Inquiry  or  inquiry-related  activity 

.520 

C409 

Ability  to  evaluate  or  critique  a  wntten 
or  oral  presentation 

<>1  A 

C004 

Ability  to  synthesize  or  summarize 

<\1 1 
.311 

C912 

Ability  to  use  ERIC  or  other  information 
retrieval  systems 

.462 

C417 

Ability  to  make  long-range  forecasts  or 
predictions 

.432 

C413 

Ability  to  conceptualize  or  "orainstorm" 
new  ideas  ' 

.424 

C015 

Ability  to  write  clearly  and/or  concisely 

.388 

C404 

Ability  to  be  creative 

.388 

Four  of  the  vaHables  (C911.  C912,  C409,  C004)  deal 

with  Wosys  of 

acquiring  or  using  information,  one  deals  specifically  with  evaluating 
information  (C409).  two  relate  to  forecasting  competencies  (C412,  C417), 
and  two  to  developing  new  ideas  (C413.  C404).   Therefore,  competency  fac- 
tor 3  is  named  ability  to  obtain  and  use  infontiation  to  fbrecast  events 
or  outcomes  or  develop  new  Ideas. 

Competency  factor  4.   This  factor  received  its  highest  loadings 
from  the  following  variables. 

C363   Knowledge  of  current  theories  of  learning, 

especially  as  they  relate  to  theories  of 

instructioi'i  ,^55^ 

-    C374   Knowledge  of  developmental  psychology  or 

the  field  of  psychology  in  general  .633 
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C543  Knowledge  of  the  role  of  the  teacher 
Including  abilities  which  normally  can 
be  expected  of  teachers  .584 

C541    Knowledge  of  Instructional  approaches 
that  might  be  incorporated  In  teaching 
or  designing  Instructional  materials  .582 

C729   Ability  to  establish  rapport  with  children 
and  obtain  their  cooperation  in  testing 
situations  .529 

C209   Knowledge  of  theory  and  techniques  for 

assessing  student  achievement  .494 

C502   Knowledge  of  printing  constraints  and 

specifications  .345 

C501    Knowledge  of  steps  Involved  in  the  mass 
production  of  curriculum  materials  (e.g., 
reproduction  and  packaging  processes)  .329 

C828   Knowledge  of  capabilities  of  local 

computer  systems  -.315 

The  first  six  of  these  nine  variables,  which  have  positive  coefficients  of 
.49  or  above,  are  consistent  with  naming  competency  factor  4  knowledge  of 
students,  teachers,  and  educational  and  psychological  processes. 

Competency  factor  5.    The  following  variables  possessed  high  load- 
ings on  this  factor: 


C61f 

Ability  to  determine  financial  resources 
necessary  to  conduct  a  program  or  pro- 
ject and  use  accounting  procedures  to 
operate  within  a  program  or  project 
budget 

.538 

C608 

Ability  to  supervise  personnel 

.487 

COOl 

Ability  to  write 

.478 

C620 

Ability  to  outline  specific  procedures 
for  working  through  a  problem 

.449 

7?. 

^^^^^^^^^^^^^^^^^^ 
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COlO   Ability  to  write  In  a  style  and  at  a 
level  appropriate  to  a  specified 

audience  444 

C209   Knowledge  of  theory  and  techniques  for 

assessing  student  achievement  -.430 

C609   Knowledge  of  effective  techniques  for 
writing  and  submitting  proposals  to 
obtain  funding  ,373 

C610   Ability  to  Identify  and  obtain  resources 

needed  to  accomplish  program  objectives  .369 

C625  Ability  to  make  progress  assessments  for 

ongoing  activities  .355 

C002  Ability  to  revise  and  rewrite  .352 

C420   Knowledge  of  legalities  related  to 

Inquiry  or  Inquiry- related  projects  .343 

C223   Ability  to  arrange  Items  In  a  format 

which  Is  easy  to  read  .342 

CS}6   Knowledge  of  and  ability  to  use  management 
and  planning  systems  such  as  PERT  (Pro- 
gram Evaluation  and  Review  Technique), 
PPBS  (Program  Planning  Budgeting 

System),  or  Critical  Path  Analysis  .339 

C417  Ability  to  make  long-range  forecasts 

or  predictions  ..323 

Of  the  14  competencies  loading  on  this  factor,  seven  (C614,  C608, 
C620,  C609,  C610,  C625  and-C616)  are  specific  administrative  competencies 
related  to  the  operation  of  research  and  research- related  projects  or 
programs.   The  competency  related  to  legal  aspects  of  conducting  such 
activities  (C420)  Is  consistent  with  competencies  related  to  the  produc- 
tion of  proposals  (C609),  formulation  of  budget-accounting  procedures  (C614) 
and  the  supervision  of  personnel  (C608).   Similarly,  the  three  writing 
competencies  which  load  on  this  factor  (COOl,  COlO,  and  C002)  are  not  sur- 
prising.  For  example,  the  ability  to  write  to  a  given  audience  (COlO)  is 
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complementary  to  competency  C609,  the  presentation  and  production  of 
proposals  to  obtain  funding.   Competency  C002— an  editing  competency- 
is  often  employed  by  senior  researchers  or  project  directors.    It  is  not 
readily  apparent  why  competency  C223,  which  is  concerned  with  the  tech- 
nical aspects  of  the  form  of  a  test  or  other  type  of  measuring  instrument, 
loads  positively  on  this  factor.    Competencies  C209  and  C417,  which  both 
load  negatively,  are  not  of  an  administrative  nature.  Consequently, 
factor  5  will  be  named  operational  administrative  competencies. 

Competency  factor  6.   The  following  variables  have  the  principle 
loadings  on  this  factor. 

C312  Ability  to  plan  effective  development 

procedures  .755 

0603  Knowledge  of  inquiry  or  inquiry- 
related  management  .629 

0604  Knowledge  of  the  role  of  inquiry  and 
inquiry-related  activities  in  education  .594 

C310   Knowledge  of  methods  for  planning  or  install- 
ing a  complete  curriculum  or  a  curriculum 
package  .471 

C611    Knowledge  of  personnel  evaluation  practices  .354 

0617  Knowledge  of  effective  techniques  of 

recruiting,  interviewing,  and  hiring  personnel     •  .337 

0607  Knowledge  of  the  organization  for  which  you 

are  working,  including  knowledge  of  its  needs, 
resources,  methods  of  operation,  etc.  .325 

The  competencies  described  above  are  needed  by  persons  in  policy  making 
and  higher  level  administrative  positions.    Two  competencies  deal  spe- 
cifically with  planning  (C312  and  C310),  one  (C603)  with  project  or 
program  management,  two  (0604  and  0607)  with  knowledge  of  the  enploying 
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organization  or  the  roles  played  by  persons  within  it,  and  two  (C611 
and  C617)  with  the  employment  and  evaluation  of  personnel.  Competency 
factor  5  and  conpetency  factor  6  are  both  related  to  administration,  with 
the  difference  being  one  of  levels  of  functioning.    Factor  6  includes 
competencies  related  to  policy  and  decision  making  on  projects  and  pro- 
grams related  to  educational  inquiry.    In  contrast,  the  competencies 
which  possess  high  loadings  on  factor  5  are  related  more  to  the  actual 
operation  of  such  projects  or  programs.    In  accord  with  this  distinction, 
competency^  factor  6  will  be  named  policy  making  and  decision  making 
competencies. 

Competency  factor  7  was  previously  named  statistical  competencies. 
Because  of  its  close  relationship  to  competency  factor  1,  this  factor  and 
variables  having  principle  lj?adings  on  itwero  discussed  earlier  along 
with  competency  factor  1. 

Iritercorrelations  among  competency  factors.  The  intercorrelations 
among  the  seven  competency  factors  are  shown  in  Table  9.    .  The  two  correla- 
tions greater  than  .15  in  absolute  value  are  both  understandable  in  view 
of  the  nature  of  the  relevant  factors.   The  highest  correlation,  that  of 
.313  between  factors  1  and  7,  is  reasonable  in  that  both  factors  are  related 
to  data  analysis  or  data  analysis  tools.  The  correlation  between  factors 
5  and  6  (.162)  is  reasonable  in  that  both  are  administrative  factors.  The 
fact  that  the  correlation  is  low  suggests  that  the  same  person  uses  both 
operational  and  policy-making  administrative  competencies  only  on  a  small 
subset  of  projects  or  programs. 
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Table  9 

Correlations  Among  Oblique  Competency  Factors:  Package  04 
Factor  l 


1 
2 
3 
4 


5 


6 
7 


139 


3 

4 

5 

6 

7 

-018 

-010 

-113 

-043 

318 

-093 

-007 

001 

108 

-064 

-110 

136 

-049 

-008 

-039 

-023 

042 

162 

-008 

-017 

Note:    Leading  decimal  points  are  omitted. 

Additional  competencies  added  to  each  rompetencv  fartor.  Upon 
conpletion  of  the  factor  analysis  of  package  04,  the  contination  of  empiri- 
cal and  logical  procedures  described  in  the  previous  section  on  data  analy- 
sis procedures  was  .  used  to  add  additional  competencies  to  those  contained 
in  the  original  seven  factors.    In  each  case,  the  variables  added  by  the 
empirical -logical  procedure  were  consistent  with  the  identification  and 
interpretation  of  each  factor  that  resulted  from  the  factor  analysis  of 
package  04.  Table  10  contains  a  final  listing  of  the  competencies  included 
under  each  of  the  competency  factors.   The  competencies  determined  by  the 
factor  analysis  of  package  04  are  listed  first,  above  the  solid  line  in 
each  column.  The  competencies  added  through  the  empirical -logical  procedure 
are  listed  below  the  solid  line  in  each  colu™,,  with  those  added  based  on 
logical  grounds  only  included  in  parentheses. 
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Table  10 

Sunmary  of  Placement  of  Competencies  Within  Competency  Factors 


i 


I 

I 


I 
I 
I 
1 
t 
i 


-RIG 


Factor ' 

Factor 

Factor 

Factor 

Factor 

Factor 

Factor 

1 

2 

3 

4 

5 

6 

7 

C102  Cl02 

C205  C105 

C206  CI 06 

C208  CI  07 

C223  C109 

C304  C115 

C408  C116 

C801  C202 

CB07  C206 

CBll  C610 

0B15  C613 

C822  C622 

C824  C629 

C825  COOl 

C826  C009 

-.C830  C103 

QB31  CI  04 

CB32  C108 

C917  Clio  ^ 

0002  Cll.l 

0)13  CI  13 

C017  CI  14 
C301       ;    CI  18 

^,.y;.-'^^i3p2i;^.,^_.;-cii^^ 


C404 
C409 
C412 
C413 
C417 
C911 
C912 
C004 
CO!  5 
ClOl 
C217 
C313 
C405 
C406 
C407 
C408 
C415 
C416 
C615 
C618 
C  904 
C906 
C915 

C401 
C903 

•  PPQI 
C  014 

tbb3 


C209 
C363 
C374 
C501 
C502 
C541 
C543 
C729 
C828 
C361 
C366 
C368 
C373 
C375 
C376 
C401 
C402 
C411 
C421 
C506 
C507 
C508 
C545 
C701 
C702 
C914 
C904 
C210 
C90$ 
W 


C209 
C223 
C417 
G420 
C608 
C609 
C610 
C614 
C616j 
C620 
C625 
,  COOl 
C002 
COlO 
C210 
C414 
C630 
C902 
C905 
'  C*915 
C018 
C628 
C615 
C627 
C601 
C602 
,v/;  C619 
C^21v;" 
;)C'623  ': 
^i;SHC6?4::-' 


C310 
C312 
C603 
C604 
C607 

ceil 

C617 
C112 
C606 
C614 
C906 
C601 
C602 


C211 
C307 
C371 
CBOl 
C810 
C812 
G819 
C828 
C201 
C304 
C305 
C306 
C366 
C804 
C808 
C823 
C827 
C834 
C308 
(C204) 
(C215) 
(C221) 
C309 
C314 
C808 
C821 

(coil) 


78 


Table  10,  continued 


mm 


Factor 
1 


Factor 
2 


Factor 
3 


Factor 
4 


Factor 
5 


Factor 
6 


Factor 
7 


C808 

C804 

C411 

C215 

C626 

C814 

C805 

C907 

C308 

C816 

C813 

C908 

C364 

C817 

C904 

C018 

C403 

C903 

C003 

C418 

C504 

• 

C918 

C006 

(C623) 

C505 

C919 

C63b 

C913 

C548 

C001 

CI  12 

C016 

(C619) 

C005 

C.212 

(C621) 

C014 

C012 

(C624) 

C015 

C204 

(C626) 

C212 

C215 

(C727) 

C305 

C221 

C728 

C112 

C308 

C821 

C204 

C309 

C901 

C215 

C314 

C918 

C221 

C403 

con 

C308 

C623 

C016 

C309 

(C624) 

C314 

(C626) 

(C403) 

C727 

C808 

C728 

C821 

C821 

C9dl 

C901 

con' 

C918 

1 


] 
] 

] 


NdtE:  The  cowpetencies  deternrined  by  the  factor  analysis  of  package  04  are 
1  is  ted  firsttoabove  the  solid  line  in  each  column .   The  competencies  added 
tjirough  the^  e^^^  below  the  son d  Vine  in 

e<a^  cpluinn^^  competency  factors  on  purely  logl- 

.caTjgi^n<te;-j^^ 


1 
1 


1 
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This  listing  represents  seven  groups  of  competencies  that  tend  to 
be  held  In  conmon  by  certain  Indlvldgals  who  are  engaged  in  exenplary 
research  and  research -related  work.   Thus,  the  analysis  has  identified 
seven  "types"  of  individuals  in  terms  of  the  competencies  they  possess. 
In  the  next  section  the  relationship  between  these  competency  factors 
and  the  12  task  factors  will  be  explored. 

Relationships  Between  Competency  Factors  and  Task  Factors 

The  two  stage  empirical  process  described  in  the  previous  section 
on  the  results  of  the  data  analysis  was  used  to  pair  the  competency  fac- 
tors and^ask  factors.   The  data  used  in  this  process  were  the  frequency 
of  occurrence  of  competencies  within  task  factors  presented  in  Table  11. 
The  first  stage  in  the  process  resulted  in  eighteen  competency  factor- 
task  factor  pairings,  with  two  additional  pairings  added  as  the  result  of 
the  second  stage.  The  results  are  listed  in  Table  12  and  discussed  under 
each  task  factor  below Each  task  factor  will  also  be  considered  as 
defining  a  broad  inquiry  or  inquiry-related  function  in  the  discussions  in 
the  remainder  of  this  paper. 


One  qualification  needs  to  be  made  concerning  these  relationships. 
Conjpetencies  required  for  the  performance  of  each  task  factor  are  not  neces 
sari ly  limited  to  those  contained  in  the  competency  factors  paired  with  the 
jacto''  here-   Although  the  most  widely  used  competencies  were  identi- 
fied in  the  factor  analysis,  inspection  of  the  frequency  of  occurrence  of 
each  competency  within  each  task  factor  (Table   11  )  shows  that  many  addi- 
tional competencies  are  used,  albeit  not  frequently. 

;^^See  definition  of  "function"  given  earlier  in  this  chapter. 
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TABLE  11 

Frequency  of  Competencies  Within  Each  Task  Factor 


TASK  FACTOR 


COMPE- 
TENCY 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

CI  01 

1 

1 

2 

1 

3 

0 

0 

1 

1 

1 

1 

6 

CI  02 

3 

6 

7 

19 

5 

0 

2 

0 

3 

5 

2 

2 

b 

CI  03 

20 

16 

36 

35 

23 

1 

8 

6 

5 

13 

12 

1  b 

3 

CI  04 

4 

5 

14 

8 

4 

0 

2 

mm 

3 

2 

b 

0 

2 

b 

3 

CI  05 

10 

12 

17 

23 

7 

0 

3 

6 

3 

6 

5 

0 

CI  06 

6 

15 

12 

23 

2 

0 

2 

1 

1 

6 

5 

1 

CI  07 

_-2 

0 

2 

7 

1 

0 

1 

2 

b 

0 

1 

0 

0 

CI  08" 

2 

8 

9 

20 

mm  w 

2 

mm 

0 

6 

0 

0 

2 

b 

5 

1 

CI  09 

3 

5 

9 

17 

1 

0 

2 

b 

0 

2 

b 

0 

0 

Clio 

9 

8 

7 

27 

2 

0 

4 

2 

b 

2 

b 

3 

1 

1 

cm 

3 

2 

3 

10 

4 

0 

2 

b 

3 

0 

2 

b 

0 

0 

CI  12 

12 

5 

5 

17 

1 

0 

2 

b 

1 

1 

3 

1 

C113 

0 

3 

6 

4 

2 

0 

2 

b 

1 

1 

0 

1 

1 

CI  14 

5 

1 

3 

4 

• 

0 

1 

3 

0 

0 

2 

b 

0 

9 

CI  15 

1 

2 

2 

la 

9 

0 

0 

0 

n 

n 

1 

0 

3 

C1 16 

2 

2 

0 

3 

0 

0 

0 

Q 

n 

0 

2 

b 

CI  17 

0 

1 
1 

1 

1 

0 

0 

Q 

1 

n 

0 

M 

0 

M 

cna 

0 

2 

4 

6 

0 

n 

Q 

2 
b 

1 

1 
1 

C1 19 

0 

1 

2 

3 

2 
b 

n 

n 

n 

1 

n 

1 
1 

C201 

5 

2  - 

] 

A 

0 

0 

1 

0 

M 

0 

M 

1 

0 

M 

C202 

11 

8 

10 

22 

1 

0 

2 

0 

3 

8 

3 

3 

C203 

2 

4 

1 

1 

0 

0 

0 

0 

1 

2 

1 

1 

C204 

12 

4 

4 

19 

1 

0 

0 

1 

0 

3 

1 

7 

C205 

2 

3 

5 

16 

2 

0 

0 

0 

0 

3 

1 

0 

C206 

1 

2 

1 

11 

0 

0 

1 

0 

0 

1 

0 

0 

C207 

10 

5 

4 

15 

0 

0 

0 

1 

2 

5 

0 

2 

C208 

6 

2 

1 

3 

0 

0 

0 

0 

0 

3 

0 

0 

C209 

3 

7 

2 

2 

1 

0 

0 

1 

0 

1 

i' 

2 

C210 

3 

6 

4 

3  . 

'  z 

0 

0 

0 

0 

0 

1 

C21.1 

2 

1 

0 

17 

0 

0 

0 

0 

0 

0 

0 

2 

C212 

0 

0 

0 

13 

0 

0 

0 

0 

0 

0 

0 

1 

C213 

1 

•  0 

0 

5 

0 

0 

0 

0 

0 

0 

0 

0 

C214 

0 

0 

0 

0 

0 

0 

0 

0- 

0 

0 

0 

1 

C215 

1 

0 

1 

6 

0 

0 

0 

0 

0 

1 

0 

2 

C216 

0 

0 

0 

4 

0 

0 

0 

0 

0 

0 

0 

0 

C217 

1 

0 

0 

11 

0 

0 

0 

0 

0 

0 

1 

0 

C218 

11 

:..2'- 

1 

4 

1 

0 

0 

1 

0 

5 

0 

4 

C219 

0 

0 

0 

2 

0 

0 

0 

0 

0 

2 

1 

1 
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TABLE  11  (cont'd) 
Frequency  of  Competencies  Within  Each  Task  Factor 


1 
I 
I 
I 


TASK  FACTOR 


COMPE- 
TENCY 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

C220 

1 

0 

0 

2 

0 

0 

0 

0 

0 

1 

0 

0 

C221 

2 

1 

2 

12 

1 

0 

1 

1 

2 

1 

1 

2 

C222 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

C223 

0 

3 

0 

8 

4 

0 

0 

0 

0 

1 

0 

2 

C301 

9 

2 

3 

6 

2 

1 

2 

4 

2 

9 

3 

1 

C302 

5 

2 

5 

9 

2 

0 

0 

0 

3 

5 

1 

0 

C303 

11 

2 

6 

10 

2 

0 

2 

4 

1 

9 

1 

1 

C304 

24 

4 

4 

9 

1 

0 

0 

1 

7 

19 

2 

2 

C305 

17 

3 

6 

11 

1 

0 

0 

3 

3 

13 

3 

2 

C306 

19 

10 

18 

12 

0 

1 

2 

1 

5 

15 

4 

1 

C307 

12 

0 

3 

14 

1 

0 

0 

1 

3 

10 

1 

0 

C308 

14 

5 

3 

12 

2 

0 

0 

6 

1 

9 

5 

1 

C309 

22 

3 

6 

21 

1 

0 

0 

3 

2 

17 

2 

2 

C310 

0 

3 

2 

2 

0 

0 

0 

0 

0 

"  1 

0 

0 

C311 

1 

1 

1 

4 

0 

0 

0 

0 

0 

1 

0 

0 

C312 

0 

6 

6 

0 

0 

0 

0 

0 

1 

0 

0 

0 

C313 

6 

2 

8 

6 

1 

1 

.1 

2 

2 

2 

0 

1 

C314 

15 

1 

2 

11 

6 

1 

1 

1 

0 

4 

0 

1 

C361 

4 

11 

10 

6 

3 

1 

0 

6 

2 

2 

1 

0 

C362 

0 

4 

4 

1 

3 

0 

0 

0 

0 

0 

1 

1 

C363 

7 

12 

11 

6 

2 

0 

0 

5 

1 

5 

2 

1 

C364 

2 

3 

3 

3 

2 

0 

1 

1 

1 

1 

1 

0 

C365 

2 

2 

2 

2 

3 

1 

0 

1 

0 

1 

2 

0 

C366 

1 

8 

2 

3 

0 

0 

0 

1 

0 

0 

3 

0 

C367 

0 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

C368 

1 

13 

7 

7 

1 

0 

2 

0 

1 

2 

4 

0 

C369 

2 

1 

2 

1 

0 

0 

0 

0 

0 

0 

0 

0 

C370 

b 

1 

0 

0 

0 

0 

0 

0 

0 

1 

2 

0 

C371 

1 

5 

0 

2 

8 

0 

0 

0 

0 

2 

3 

2 

-6372 

0 

5 

r 

0 

0 

0 

0 

0 

0 

0 

0 

0 

C373 

4 

2 

1 

3 

0 

0 

0 

0 

0 

3 

0 

0 

C374 

13 

12 

10 

9 

4 

0 

.1- 

8 

0 

7 

8 

2 

C375 

0 

4 

2 

1 

0 

0 

0 

2 

1 

0 

0 

0 

C376 

4 

16 

11 

4 

2 

0 

0 

1 

0 

1 

0 

C377 

0 

d 

0 

0 

3 

0 

0 

0 

0 

0 

2 

C378 

0 

1 

2 

4 

4 

0 

0 

0 

0 

0 

1 

2 

C4d1 

31 

24 

38 

24 

25 

5 

3 

28 

9 

19 

11 

6 

C402 

4 

11 

12 

3 

4  - 

0 

1 

7 

0 

2 

1 

1 
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TABLE  11  (cont'd) 
Frequency  of  Competencies  Within  Each  Task  Factor 


TASK  FACTOR 


COMPE- 
TENCY 

1 

2 

3 

4 

c 
9 

6 

7 

8 

9 

1  n 
10 

1 1 

1  o 
Id 

C403 

1 

3 

1 

2 

1 

0 

0 

1 

0 

0 

1 

1 

C404 

2 

3 

2 

0 

5 

0 

0 

1 

2 

2 

1 

1 

C405 

18 

13 

20 

12 

18 

3 

2 

11 

7 

4 

4 

2 

C406 

10 

1 

1 

1 

5 

0 

0 

8 

0 

6 

0 

0 

C407 

11 

7 

12 

6 

7 

1 

0 

6 

3 

5 

2 

4 

C408 

2 

2 

2 

4 

4 
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TABLE  11  (cont'd) 
Frequency  of  Competencies  Within  Each  Task  Factor 
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Frequency  of  Competencies  Within  Each  Task  Factor 
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TABLE  11  (cont'd)  ' 
Frequency  of  Competencies  Within  Each  Task  Factor 
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Table  12 

Competency  Factor  -  Task  Factor  Pairings 


Task  Factors 

Competency  Factors 

1 

1.  7* 

2 

2,4,5 

3 

2,  5,  6* 

4 

c 

5 

1.  3 

6 

3 

7 

2,  5 

8 

3 

9 

5 

10 

11 

5,  6 

12 

4 

^his  competency  factor  is  one  of  two  identified 
by  the  second  stage  of  the  identification  process 
described  in  the  section  on  data  analysis  procedures. 


Task  factor  1.   Designing  research  studies  and  conducting  and  inter- 
preting data  analyses.  The  competency  factors  related  to  this  task  factor 
are  No.l.  data  collection,  processing,  analysis,  and  presentation  competen- 
cles.  and  No . 7 .  stati sti cal  conpetenci es .   A  review  of  the  competencies  that 
make  up  these  two  competen<y  factors-  indicates  that  T)ersons  involved  in  this 
function  (task  factor)  use  sicills  of' data  processing,  data  analysis,  informa- 
tion presentation  and  instrument  construction,  and  knowledge  of  statistics, 
experimental  controls  and  computer-capabilities. 


I 


87 
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{  . 

1  Task  factor  2.   Developing  instructional  materials.    The  competency 

factors  related  to  this  task  factor  are  No.  2»  evaluation  cowpetencies; 
No.  4.  knowledge  of  students,  teachers,  and  educational  and  psychological 
processes ;  and  No.  5.  operational  administrative  skills.   A  review  of  the 
1  competencies  which  form  these  three  competency  factors  shows  that  the  com- 

j  petencies  are  logically  related  to  the  function  of  developing  instruc- 

tional materials.    Evaluation  skills  are  obviously  relevant  to  the  field 
and  product  testing  aspects  of  development.   Knowledge  about  students, 
teachers,  learning  theories,  and  educational  processes  is  obviously  nec^- 
I  sary  to  the  person  developing  instructional  materials.  Skills  in  adminis- 

I  tering  a  project  and  supervising  work  of  other  persons  are  also  relevant 

since  development  seems  typically  to  be  a  team  activity  requiring  coordi- 
I  nated  efforts  of  many  persons.  ^ 

Task  factor  3.   "First-level"  administration  of  inquiry  and  inguiry- 
I  related  projects 'and  activities.  ^  Competency  factors  No.  2,  evaluation  com- 

petencies;  No.  5,  operational  admi  ni  strati  ve  ski  lis:  and  No.  6,  policy  mak- 
'  .  ing  and  decision  making  competencies  are  related  to  this  task  factor.  Study 

I  of  these  competency' factors  shows -the  inclusion  of  the  obviously  relevant 

clusters  of  administrative  competencies  such  as  budgeting,  personnel  super- 
I  vision,  specification  of  work  tasks  and  procedures,  resource  allocation,  per- 

sonnel hiring  and  evaluation,  and  management  planning  systems  such  as  PERT. 
It  is  not  surprising  that  evaluation  competencies  are  included  in  view  of 


I  their  contribution  to  administration  through  providing  informati 

deciisiidn  making. 


88 

Task  factor  4.   Conducting  evaluations  and  constructing  and  using 
data  collection  Instruments.   Competency  factor  No.  2,  evaluation  conpeten- 
cles,  is  related  to  this  function,  which  would  be  expected  in  view  of  the 
similar  nature  of  the  competency  and  task  variables.   Included  in  this 
competency  factor  are  competencies  such  as  planning  evaluations,  specify- 
ing objectives,  constructing  attitude  scales,  putting  numerical  informa- 
tion in  written  form,  and  knowledge  of  evaluation  and  measurement. 

Task  factor  5.   Diffusing  information  and  products.  Competency 
factors  No.  1.  data  collection,  processing,  analysis,  and  presentation 
competencies,  and  No.  3.  ability  to  obtain  and  use  information  to  fore- 
cast events  or  outcomes  or  develop  new  ideas  are  related  to  this  task 
factor.   The  latter  competency  factor  emphasizes  dissemination  skills 
such  as  using  library  search  processes,  writing  clearly  and  concisely,  and 
summarizing  and  synthesizing  ideas.  The  most  relevant  skills  on  the  first 
competency  factor  are  presentation  competencies,  such  as  presentation  of 
data,  describing  or  elaborating  in  writing,  and  putting  numerical  informa- 
tion into  verbal  form. 

Task  factor  6.   Developing  and  operating  information  storage  and 
retrieval  systems.   Competency  factor  Nn.  3,  ability  to  obtain  and  use 
information  to  forecast  events  or  outcomes  or  develop  new  ideas,  is  related 
to  this  task  factor.  This  competency  factor  includes  library  search  skills, 
the  ability  to  critique  written  presentatiorKS,  the  ability  to  summarize  and 
synthesize,  knowledge  of  and  ability  to  use  Information  retrieval  systems, 
and  the  ability  to  write  clearly  and  concisely.   All  these  competencies 
are  clearly  relevant. 
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Task  factoV  7.    Evaluating  Inquiry  and  inquiry- related  proposals 
and  monitoring  funded  prolects.   Two  competency  factors  related  to  this 
task  factor  are  No,  2.  evaluation  competencies,  and  No.  5.  operational 
administrative  skills.  Many  evaluation  competencies  obviously  come  into  play 
in  evaluating  proposals.   Administrative  competencies  useful  in  monitoring 
funded  projects  include  ability  to  make  progress  assessments  for  ongoing 
activities,  knowledge  of  legalities  related  to  project  management,  and 
knowledge  and  ability  to  use  management  systems  such  as  PERT. 

Task  factor  8.    Searching,  reading,  and  reviewing  the  literature. 
Coirpetenoy  factor  No.  3.  ability  to  obtain  and  use  information  to  forecast  . 
events  or  outcomes  or  develop  new  ideas,  is  related  to  this  task  factor. 
Several  competencies  on  competency  factor  3  which  are  logically  related  to 
the  process  of  obtaining  information  from  the  literature  include  the  follow- 
ing:  ability  to  use  library  research  techniques,  ability  to  critique  writ- 
ten presentations,  ability  to  summarize  or  synthesize,  and  ability  to  use 
information  retrieval  systems. 

Task  factor  9.    Designinq  and  maintaining  computer  systems  and  writing 
cowputeV*  programs.   Competency  factor  No.  5,  operational  administrative  skills, 
is  related  to  this  task  factor.   Skills  in  budgeting,  personnel  supervision, 
work  task  and  procedures  specification,  resource  allocation,  writing,  and 
knowledge  of  proposal  preparation  are  included  on  competency  factor  5.    It  is 
not  unexpected  that  these  administrative  skills  are  employed  in  designing 
computer  systems.  ^ 

Task  factor  10.  >  Unnamed.    Rpraii«tP  tasks  within  this  factor  did  not 
appear  logically  connected  in  any  way,  no  attempt  v^iis  made  to  place  competency 
factors  With  task  factor  10. 
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Task  factor  11,    "Second«»1eve1"  administration  of  Inquiry  and  Inquiry- 
related  projects  and  activities.    Competency  factors  related  to  this  function 
are  No.  5>  operational  administrative  skills^  and  No.  6,  policy  making  and 
decision  making  competencies.    Both  the  task  factor  and  the'two  competency 
factors  are  clearly  focused  upon  administration.    Among  the  administrative 
competencies  listed  under  these  factors  are  skill  in  project  budgeting; 
personnel  supervision;  specification  of  work  tasks  and  procedures;  resource 
allocation;  recruiting,  hiring  and  evaluating  personnel;  making  progress 
assessment  of  ongoing  activities;  ability  to  make  long-range  forecasts;  know- 
ledge of  project  or  program  management;  knowledge  of  role  of  inquiry  and 
inquiry- related  activities  in  education;  and  knowledge  of  management  and 
planning  systems  such  as  PERT, 

Task  factor  12,    Developing  and  scoring  tests.    Competency  factor 

ft 

No.  4>  knowledge  of  students,  teachers,  and  educational  and  psychological 
processes,  is  related  to  this  task  factor.   Knowledge  of  learning  theories 
and  developmental  psychology,  fcnowledge  of  teacher  roles,  knowledge  of 
instructional  approaches,  knowledge  of  theory  and  techniques  for  assess- 
ing student  achievement,  and  ability  to  obtain  cooperation  of  students  in 
testing  situations  are  obviously  competencies  which  are  relevant  to  t^is 
task  factor, 

.         Rel  ationships  Between  RDDE  and 
Inquiry  and  Inquiry-related  Functions  Identified  Through  These  Data 

The  basic  objective  of  this  studly  was  to  i;^dentify  the  functions,  tasks 
and  competencies  required  in  exemplary  educational  research  and  research- 
Te1ated;activities.   The  results  of  the  data  analysis  reported  in  previous 
secti ons  of  thi s  chapter  have  i  denti fied  clearly  tasks  and  ampetencies 
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required  in  such  activities.  The  eleven  interpretable  task  factors  also 
identify  eleven  broad  functions  (groups  of  tasks)  wh'ich  are  performed  by 
inquii^y  and  inquiry-related  personnel  in  the  field  of  education.  A 
discussion  of  these  functions  and  their  relationship  to  the  commonly  used 
rubrics  of  research,  development,  diffusion,  and  eval uation  is  contanned 
in  this  section. 


of  activities  or  tasks  which  taken  together  lead  to  the  attain- 
ment of  a  particular  inquiry  goal."      Examples  of  functions  were 
listed  as  research,  development,  diffusion,  and  evaluation  (RDDE)~four 
activities  which,  on  a  priori  grounds,  have  been  thought  of  as  collectively 
comprising  the  spectrum  of  inquiry  and  inquiry- related  activities  in  edu- 
cation.   It  was  the  intent  In  this  study  to  Ignore  the  RDDE  rubrics  and 
let  the  data  analysis  suggest  functions  which  could  be  defended  on  empiri- 
cal grounds.    However,  it  was  also  the  intent  to  try  to  relate  functions 
Identified  through  the  data  analysis  with  the  more  commonly  used  categories 
of  RDDE.   The  attempt  to  draw  such  relationships  appears  below,  in  Table  13. 


A  "function"  was  defined  for  this  study  as  a  "broad  range 
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TABLE  13 


Interpretation  of  Relationships  Between  Task  Factors 
and  Research,  Development,  Diffusion,  and  Evaluation 


Task  Factor 

Is  thought  to  be  related  to 
Research  (R),  Development 
(De),  Diffusion  (Di),  or 
Evaluation  (E) 

1 

Designing  research  studies  and  con- 

uuui#iiii|  anu  inuerprctiny  uaLa  andiySGS 

R  (E) 

2 

Developing  instructional  materials 

De 

3 

"First-level"  administration  of  in- 
quiry and  Inquiry- related  projects 
and  activities 

4 

Conducting  evaluations  and  construct- 
mg  and  using  data  collection  instru- 
ments 

E  (R) 

5 

Diffusing  Information  and  products 

01 

6 

Developing  and  operating  Information 
storage  and  retrieval  systems 

(DI) 

7 

Evaluating  Inquiry  and  inquiry- 
related  proposals  and  monitoring 
funded  projects 

(E) 

8 

Searching,  reading,  and  reviewing 
the  literature 

(R) 

9 

Designing  and  maintaining  computer 
systems  and  writing  computer  pro- 
grams 

10 

Unnamed 

11 

"Second-level"  administration  of 
Inquiry  and  Inquiry- related  projects 
and  activities 

12 

Developing  and  scoring  tests 

NOTE:   Parenthetical  entries  In  the  column  at  the  right  denote 
emphasis. 
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The  relationships  between  the  categories  of  RDDE  and  the  12  func- 
tions (task  factors)  presented  in  Table  13  are  viewed  as  tentative;  the 
relationships  are  not  data-based  but  rely  instead  on  logical  interrelation- 
ships which  seem  defensible.   Others  may  find  compelling  arguments  for 
proposing  different  interrelationships.    Such  efforts  seem  appropriate  since 
the  intent  here  is  to  suggest  probable  relationships  rather  than  to  argue 
that  all  defensible  relationships  have  been  included.    With  this  proviso,  the 
relationships  shown  in  Table  13    are  discussed  briefly  below. 

The  function  of  designing  research  studies  and  conducting  and  inter- 
preting data  analyses  (task  factor  1)  is  clearly  a  part  of  research  as  we 
have  comnonly  used  the  term.    In  addition,  evaluation  is  dependent  upon 
data  analysis  and  interpretation  and  therefore  overlaps  part  of  this  function 
as  well.   Although  data  may  be  analyzed  and  interpreted  as  part  of  a  develop- 
ment or  diffusion  effort,  it  does  not  follow  that  this  function  is  therefore 
development  or  diffusion.    In  development,  data  analysis  and  interpretation 
would  normally  occur  only  during  product  testing  or  field  testing  activities, 
which  are  clearly  evaluation  by  another  name.    In  diffusion,  data  analysis 
and  interpretation  might  take  place  in  activities  such  as  market  analysis; 
however,  this  is  simply  research,  used  in  this  instance  to  provide  informa- 
tion for  use  by  diff users. 

1  g 

RDDE  can  each  be  applied  to  one  another.    For  example,  one  can  do 
research  on  diffusion,  development,  or  evaluation  processes.    One  can  evalu- 
ate research,  development,  or  diffusion  efforts.    Results  of  research,  evalua- 
tion, and  development  can  all  be  diffused.   Instructional  materials  can  be 
developed  to  teach  research,  evaluation,  or  diffusion.    However,  such  inter- 
relationships seem  to  the  authors  to  be  conceptually  sterile  and  are  not 
Included  In  the  emphases  shown  in  Table  13. 

m 
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The  function  of  developing  instructional  materials  (task  factor  2) 
obviously  includes  most  of  what  has  been  referred  to  as  development  in  the 
commonly  used  rubric  of  RDDE  and  requires  little  comment  here.    The  fact 
that  other  development  efforts  (e.g.,  the  development  of  organizational 
or  staffing  plans)  is  not  included  is  probably  a  simple  reflection  of  the 
fact  that  development  of  instructional  materials  currently  occupies  most 
of  the  attention  of  the  relatively  small  cadre  of  educational  developers. 

The  administrative  functions  (task  factors  3  and  11)  cut  across  all 
four  categories  of  RDDE;  administration  at  both  levels  is  required  in  pro- 
jects or  programs  of  all  four  types.   However,  administering  a  research 
project  is  not  viewed  here  as  research  fier  se,  but  as  a  function  which  facili- 
tates research;  parallel  reasoning  applies  to  the  administration  of  DDE  acti- 
vities.  Therefore,  administration  is  viewed  here  as  an  activity  that  is 
neither  RDD  nor  E  but  is  requisite  for  successful  accomplishment  of  any  such 
activities. 

The  function  of  conducting  evaluations  and  using  data  collection  instru- 
ments (task  factor  4)  is  clearly  the  E  of  RDDE.    In  addition,  data  collection 
instruments  ai^  essential  in  research  activities  and  research  is  therefore 
viewed  as  overlapping  partially  with  this  function.   As  with  task  factor  1, 
this  function  ma^y  be  useful  in  development  or  diffusion  efforts,  but  only  to 
the  extent  that  evaluation  or  research  is  a  necessary  part  of  such  efforts. 

Diffusing  infonnation  and  products  (task  factor  5)  is  clearly  the 
same  function  that  has  been  described  in  the  literature  as  diffusion.  It 
should  be  noted,  however,  that  the  activities  which  compose    this  function 
are  for  the  most  part  dissemination  activities  and  do  not  include  other 
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activities  such  as  demonstration  or  adoption  which  are  typically  viewed 
as  part  of  the  diffusion  process  (e.g.,  discussions  of  diffusion  by  Clark 
and  6uba^^).   It  may  be  that  diffusion  does  include  such  activities  and 
the  data  analyzed  here  are  inaccurate  because  the  agencies  and  individuals 
interviewed  are  not  engaged  in  the  full  range  of  diffusion  activities  which 
would  have  been  found  had  other  agencies  or  individuals  been  selected.  Con- 
versely, It  may  be  that  the  data  reflect  reality  and  activities  such  as 
demonstration  and  adoption  are  Important  and  viable  only  in  the  minds  of 
academicians  and  not  in  the  real  world  of  diffusion.   The  data  presented  here- 
in do  not  allow  resolution  of  this  question. 

Developing  and  operating  information  storage  and  retrieval  systems 
(task  factor  6)  is  viewed  as  one  type  of  diffusion.    It  is  clearly  not  research, 
evaluation,  or  development,  although  it  ma^y  be  useful  in  any  of  these  activi- 
ties.  The  necessary  activity  of  searching  out  information  for  input  into  such 
systems  may  be  a  type  of  research;  however,  such  activities  are  subsumed 
under  task  factor  8,  searching,  reading,  and  reviewing  the  literature.  This 
function  (task  factor  8)  may  also  be  useful  in  development,  diffusion,  or 
evaluation,  but  it  seems  reasonable  to  interpret  it  as  a  type  of  research 
function  used  In  these  activities-  rather  than  DD  or  E  per  se. 

Although  there  is  an  element  of  evaluation  in  task  factor  7,  evaluating 
Inquiry  and  inqui ry- related  proposals  and  monitoring  funded  projects,  this 
should  not  obscure  the  fact  that  this  is  largely  a  function  of  personnel  in 

^^Clark,  D.  L.  and  Cuba,  E.  6.    An  examination  of  potential  change 
roles  In  education.    Paper  presented  at  a  Seminar  on  Innovation  in  Planning 
School  Curricula    at  Aerliehouse,  Virginia,  Oct. 2  -  4,  1965. 
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funding  agencies  or  persons  temporarily  helping  funding  agencies  reach  eval- 
uative judgments  and  is  not  a  routine  function  of  the  professional  evaluator. 

Designing  and  maintaining  computer  systems  and  writing  computer  pro- 
grams (task  factor  9)  seems  a  function  related  to  but  independent  of  RDDE. 
It  is  used  most  frequently  in  R  and  E  but  is  not  in  and  of  itself  research 
or  evaluation,  even  though  it  serves  as  a  tool  in  both.   The  same  logic 
applies  to  task  factor  12 »  developing  and  scoring  tests.    This  function 
is  critical  to  much  research  and  evaluation  in  that  tests  are  among  the 
most  common  data  collection  instrurrents;  however,  this  does  not  make  measure- 
ment research  or  evaluation. 

The  attempts  to  relate  the  12  functions  (task  factors)  to  RDDE  should 
not  be  interpreted  as  meaning  that  RDDE  are  viewed  as  preferable  (or  even 
valid)  categories  of  inquiry  and  inquiry- related  activities.   The  11  inter- 
pretable  task  factors  are  proposed  here  as  more  meaningful  categories  for 
use  by  persons  concerned  with  training  inquiry  and  inquiry- related  personnel. 
The  attempt  to  relate  these  functions  to  RDDE  is  merely  an  attempt  to  assist 
per-sons  who  are  more  familiar  with  that  rubric  to  Interpret  or  use  the  re- 
sults of  this  studly. 


i 
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Summajry 

Data  obtained  In  Interviews  with  persons  engaged  In  exemplary 
educational  research  and  research-related  work  were  categorized 
into  69  task  categories  and  226  categories  of  competencies  employed 
in  performing  these  tasks.      Inter-judge  reliabilities  of  transfer 
of  interview  data  to  written  records  and  categorization  of  these  data  were 
computed.    Factor  analysis  was  used  to  isolate  12  task  factors  or  functions. 
Factor  analysis  and  additional  empirical -logical  procedures  were  used  to 
identify  seven  competency  factors.    Relationships  between  task  factors  and 
competency  factors  were  identified.   Interpretations  of  these  factors  and 
relationships  between  the  task  factors  and  the  commonly  used  categories 
of  RDDE  are  discussed. 

It  is  recommended  that  the  functions,  tasks,  competencies  and  their 
interrelationships  identified  in  this  study  be  viewed  as  tentative  and  in 
need  of  validation.   Limitations,  such  as  the  size  of  the  sample  (116), 
when  viewed  in  the  light  of  the  wide  range  of  tasks  and  competencies  Identi- 
fied,'^must  be  considered.   Should  further  validations  be  conducted,  the 
pilot  work  reported  herein  should  provide  a  basis  for  the  development  of 
data  collection  procedures  that  can  be  focused  on  precise  functions,  tasks 
and  competencies  and  thus  collect  more  data  about  each  than  was  possible 
in  this  attempt  to  depict  the  domain. 
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CHAPTER  III 


DEVELOPMENT  OF  A  SELF-REPORT  INSTRUMENT  FOR  SELECTED  SKILLS 
AND  KNOWLEDGE  IN  EPIJCATIONAL  RESEARCH,  DEVELOPMENT, 
DIFFUSION,  AND  EVALUATION 
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DEVELOPMENT  OF  A  SELF-REPORT  INSTRUMENT  FOR  SELECTED  SKILLS 
AND  KNOWLEDGE  IN  EDUCATIONAL  RESEARCH,  DEVELOPMENT, 
DIFFUSION,  AND  EVALUATION 

There  is  a  need  for  satisfactory  instruments  designed  to  assess 
the  competence  level  of  educational  research  and  research-related 
personnel.    Such  measures  are  needed  to  enable  both  research -training 
and  research-producing  rgencies  to  assess  competencies  and  deficiencies 
of  trainees  and  employees  as  a  basis  for  planning  future  pre-service 
or  in-service  training  programs.    In  recognition  of  this  need,  the 
current  Task  Force  project  included  two  objectives  relating  to  the 
development  of  relevant  multi -dimensional  competence  instruments  and 
assessment  procedures.   One  objective  (Objective  2.1)  related  to  the 
development  of  mastery  tests  of  certain  competencies  in  educational 
research,  development,  diffusion,  and  evaluation  (RDDE);  the  development 
of  these  tests   is  reported  in  the  next  chapter  of  this  report. 
The  second  related  objective  (2.2)  was  stated  as  follows: 


The  development  of  these  measures  was  undertaken  for  four  major 
reasons:    (1)  the  development  of  pilot  tests  in  Objective  2.1  was 
viewed  as  preliminary,  requiring  extensive  and  lengthy  tiryout  and  norming 
of  the  tests  before  they  could  be  used  and  interpreted  confidently;  there- 
fore, the  questionnaires  were  viewed  as  valuable  interim  measures; 


To  design  and  develop  questionnaires  to 
assess  RDDE  students'  and  workers'  perceptions 
of  their  existing  competencies  and  training 
needs. 
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(2)  self-report  questionnaires  could  be  used  in  a  variety  of  situations 
where  administration  of  mastery  tests  might  not  be  possible;  (3)  ques- 
tionnaires could  be  used  more  readily  than  mastery  tests  to  probe 
areas  where  trainees  or  employees  perceived  a  need  for  additional 
training;  and  (4)  it  was  demonstrated  in  earlier  work  of  the  Task  Force 
(Technical  Paper  No,  19)  that  self-reports  of  competence  in  RDDE  were 
reliable  and  apparently  not  influenced  by  any  acquiescence  set  on  the 
part  of  respondents.    Procedures  used  by  the  Task  Force  in  developing 
questionnaires  to  assess  RDDE  competencies  and  perceived  training 
needs  are  reported  in  this  chapter. 

Instrument  Development  Procedures  and  Specifications 

Discussions  of  specifications  for  the  questionnaires,  the 
content  included,  and  steps  in  the  development  process  are  included 
in  this  section. 

Specifications  for  the  Instrument 

The  Task  Force  proposed  that  the  content  of  the  questionnaires 
be  drawn  from  the  analysis  of  data  yielded  in  relation  to  Objective  1 
of  the  current  project  (see  Chapter  II  above).      Until  those 
data  were  analyzed,  it  was  impossible  to  determine  whether  it  would 
be  best  to  develop  (a)  multiple  questionnaires  to  measure  self-reports 
of  competencies  within  the  categories  of  RDDE,  (b)  multiple  question- 
naires to  measure  self-reports  of  competencies  within  other  categories 
of  competencies  identified  through  the  analysis  of  Objective  1  data, 
or  (c)  a  single  questionnaire  including  competencies  that  cut  across 
categories.    Subsequently,  a  factor  analysis  of  Objective  1  data 
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yielded  12  factors  relating  to  research  and  research-related  tasks; 
each  factor  subsumed  numerous  competencies •    For  the  purpose  of 
developing  the  questionnaires,  the  12  task  categories  were  collapsed 
as  much  as  possible  (as  described  in  a  later  section  of  this  chapter) 
to  yield  seven  task  categories.    Examination  of  the  seven  task 
categories  and  competencies  within  categories  led  to  the  following 
four  decisions: 

1.  It  was  decided  that  the  new  task  categories  should  be  used 
as  the  basis  for  instrument  construction  rather  than  the  four  categories 
of  RDDE,  which  lacked  empirical  support  in  the  data  analysis. 

2.  It  was  decided  that  seven  rather  specific  instruments,  som? 
of  which  would  include  relatively  few  competencies,  would  be  useful  only 
for  extremely  specific  training  programs  or  activities,  of  which  few 
real -world  examples  could  be  found. 

3.  t4ost  training  programs  cut  across  several  of  the  task 
categories  and  their  directors  would  be  interested  in  self -reports  of 
competencies  in  several  of  the  areas.   (For  example,  each  of  the  three 
new  training  consortia  funded  by  the  OE  Research  Training  Branch 
offers  training  in  several  of  the  task  categories.) 

4.  It  was  decided  to  develop  one  "instrument"— an  item  pool 
accorr^anied  by  directions  for  administration—that  could  either  be 
used  as  a  single  broad  instrument  or  subdivided  in  numerous  ways 

to  allow  flexible  use  by  trainers  and  employers. 

Based  on  these  decisions,  the  following  specifications  for 
the  instrument  were  established:    (1)  the  instrument  would  include  a 
general  set  of  directions  which  would  be  used  with  the  total  instrument 
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or  any  subdivision  of  it;  (?)  the  format  of  tht^  instrument  (with  the 
exception  of  refinements  to  Increase  clarity  and  the  p.dditlon  of  self- 
reports  of  training  needs)  would  follow  the  format  reported  in  Technical 
Paper  No.  19;  (3)  the  instrument  would  include  a  pool  of  Items  selected 
to  measure  important  competencies  in  each  task  category;  and  (4)  each 
item  would  be  keyed  to  each  factor  (and,  if  possible,  to  the  categories 
of  RDDE  as  well)  to  facilitate  subdividing  the  instrument  by  selecting 
items  relevant  to  specific  categories. 

Content  of  the  Instrument 

Data  from  Objective  1  provided  the  base  from  which  competencies 
were  selected  for  inclusion  in  the  instrument.    The  Objective  1  data 
analysis  resulted  in  12  task  factors  which  appeared  to  provide  the 
most  meaningful  grouping  of  tasks  performed  in  conducting  research  and 
research -related  activities.   Names  of  the  ]Z  factors  are  listed  below. ^ 

Factor  1:    Designing  research  studifis  and  conducting  and 
interpreting  data  analyses. 

Factor  2:    Developing  instructional  materials. 

Factor  3:    "First-level"  administration  of  inquiry  and  inquiry- 
related  projects  and  activities. 

Factor  4:    Conducting  evaluations  and  constructing  and  using 
data  collection  instruments. 


Generation  of  these  factors  and  their  logical  relationships 
with  earlier  a  priori  categories  of  RDDE  are  discussed  in  Chapter  II. 
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Factor  5:    Diffusing  information  and  products. 
Factor  6:    Developing  and  operating  information  retrieval 
systems. 

Factor  7:    Evaluating  inquiry  and  inquiry- related  proposals 
and  monitoring  funded  projects. 

Factor  8:    Searching,  reading,  and  reviewing  the  literature. 

Factor  9:    Designing  and  maintaining  computer  systems  and 
writing  coatputer  programs. 

Factor  10:  No  name  given  to  this  factor  since  its  interpreta- 
tion was  unclear. 

Factor  11:  "Second-level"  administration  of  inquiry  and  inquiry- 
related  projects  and  activities. 

Factor  12:  Developing  and  scoring  tests. 

For  the  purpose  of  constructing  the  present  instrument,  the  twelve 
factors  were  combined  logically  to  form  seven  task  factor  categories 
which  appeared  to  be  probable  training  specializations.    These  seven 
task  categories  are  listed  below,  with  the  corresponding  number  from 
the  previous  list  of  factors  shown  in  parentheses. 

Category  1 :    Designing  research  studies  and  conducting  and 
interpreting  data  analyses  (1). 

Category  2:    Developing  instructional  materials  (2). 

Category  3:    Administering  inquiry  and  inquiry-related  projects 
and  activities  (3  and  11). 

Category  4:    Conducting  evaluations  and  constructing  and  using 
data  collection  instruments  (4  and  12). 

If8 
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Category  5:   Seeking  information,  storing  and  retrieving  infor- 
mation, and  diffusing  information  and  products  (5,  6  and  8). 

Category  6:   Evaluating  inquiry  and  inquiry-related  proposals 
and  monitoring  funded  projects  (7). 

Category  7:   Designing  and  maintaining  computer  systems  and 

writing  computer  programs  (9). 

.   Factor  10  was  excluded  because  of  its  uninterpretability. 

The  2Z6  competencies  (both  skills  and  knowledge)  identified  in  the 

Objective  1  interview  sample  (see  Chapter  II)  were  tabulated  according  to 
the  sev^jn  task  categories  above.    Each  competency  was  listed  under  each 
task  category  in  which  the  Interviewee  indicated  that  he  used  the  coffpe- 
tency.  Within  each  category,  frequencies  were  examined  and  the  competen- 
cies that  appeared  most  frequently  were  selected  for  possible  inclusion 
In  the  Instrument.   The  rationale  here  was  to  select  for  assessment  those 
competencies  that  were  often  found  to  be  required  in  the  work  of  persons 
engaged  in  research  and  research-related  activities  viewed  as  exemplary. 

It  should  be  noted  that  the  list  of  competencies  resulting  from 
the  application  of  the  frequency  criterion  contained  several  competencies 
which  might  be  regarded  by  some  persons  as  inappropriate— or  at  least  non- 
specific—to RDDE  activities.    (Knowledge  of  printing  constraints  and  mass 
production  techniques  for  curriculum  materials,  and  knowledge  of  funding 
sources  for  proposals  might  be  cited  as  examples.)    It  was  felt  by  the 
Task  Force  staff,  however,  that  since  these  competencies  were  used  fre- 
quently by  a  sample  of  persons  engaged  in  exemplary  RDDE  work,  they  could 
not  be  eliminated  from  consideration  as  long  as  they  met  the  criteria 
listed  below  which  were  applied  to  the  other  competencies  in  the  list.  To 
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have  dismissed  them  oii  the  basis  of  our  own  notions  of  what  is  or  is  not 

relevant  to  the  conduct  of  research  in  education  would  have  done  grave 

injustice  to  the  empirical  nature  of  the  study. 

Each  competency  selected  on  the  basis  of  frequency  of  occurrence 

2 

was  then  screened  on  the  basis  of  the  following  three  criteria: 

1.  Is  the  competency  trainable;  i.e.,  does  it  include  know- 
ledge or  a  skill  which  can  be  conmunicated  through  systematic  training 
or  does  it  represent  a  personal  characteristic  (e.g.,  "ability  to  be 
creative")  that  is  unlikely  to  be  enhanced  through  training? 

2.  Is  the  competency  the  end  product  of  substantive  training 
in  another  field  (e.g.,  "knowledge  of  developmental  psychology")? 

3.  Is  the  competency  important  or  significant;  i.e.,  is  it 
something  which  cannot  be  so  easily  communicated  as  to  be  trivial  in 
a  training  program  (e.g.,  "ability  to  pay  attention  to  detail")? 

If  the  competency  failed  to  meet  Criterion  1  above,  it  was 
excluded  since  the  purpose  in  developing  the  instrument  is  to  provide 
a  useful  tool  to  persons  concerned  with  identifying  competencies 
where  additional  training  is  needed.    The  competency  was  also  excluded 
if  Criterion  2  was_met,  since  it  seems  unreasonable  to  expend  scarce 
resources  to  provide  training  for  competencies  for  which  well- 
established  training  programs  already  exist  and  from  which  persons 
possessing  the  competencies  could  be  recruited.    The  third  criterion 


^These  criteria  are  modified  from  those  listed  in  the  original 
proposal:   criticality,  gene raliz ability,  md  relevance.   The  la«er  ^as 
ensCred  by  selecting  from  a  relevant  data  base;  the  first  was  subsumed 
under  criterion  throe  above;  and  the  second  was  viewed  as  an  abstraction 
of  the  froquency  criterion  employed  above.   Criteria  1  and  Z  listed 
above  wero  added  to  those  originally  proposed. 
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was  included  in  recognition  of  the  fact  that  many  of  the  competen- 
cies, although,  important  in  the  conduct  of  an  inquiry  activity  might 
well  be  trivial  in  a  training  sense.    For  example,  it  seems  sufficient 
to  inform  potential  research  managers  that  one  must  "pay  attention  to 
administrative  detail"  without  contemplating  the  establishment  of 
courses  in  "paying  attention  to  details  " 

The  application  of  the  above  criteria  within  each  category 
resulted  in  a  list  of  competencies  with  108  entries,  distributed  across 
task  categories  as  shown  in  Table  14  below.  Because  of  the  small  number 

of  items  in  categories  two,  five  and  six,  additional  competencies  were 
added  to  provide  a  basis  for  more  reliable  measurement  within  those 

categories.     This  was  done  by  going  back  to  the  data  for  those 
categories  and  selecting  competencies  which  seem  important  on  logical 
grounds  but  which  fell  below  the  original  frequency  cut-off  point.  Addi- 
tional competencies  chosen  on  this  basis  are  show/i  in  the  third  column 
of  Table  14. 

Table  14 

Number  of  Competencies  for  Each  Task  Category  Included  in  the  Instrument 

Total  N  of 


Task 
Category 

N  of  Competencies, 
Based  on  Original 
Application  of  Criteria 

N  of  Competencies 
Added  to  Increase 
N  in  Some  Categories 

Competencies 
in  Final  Instrument 
in  Each  Category 

1 

32 

2 

34 

2 

8 

8 

16 

3 

17 

2 

19 

4 

26 

2 

28 

5 

8 

5 

13 

6 

5 

1 

6 

7 

12 

0 

12 

Some  of  these  competencies  also  appeared  in  other  task  categories, 

— Ha       *^  resitltlna  In  aratultous  additions  in  cateaories  one,  three  and  four. 

1.  I 
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It  should  be  noted  that  the  total  number  of  competency  entries 
in  Tab1e14  (128)  is  not  the  number  of  different  competencies  included. 
Many  competencies  appear    under  more  than  one  task  factor  and  therefore 
are  counted  more  than  once  in  the  totals  above.    The  total  number  of 
discrete  competencies  generated  through  the  procedures  outlined  above 
is  75.   These  competencies  can  be  identified  by  cro?.s-referencing  the 
competency  number  for  each  Item  in  Appendix  H»  with  the  list  of 
competency  numbers  in  Appendix  D. 

Development  of  the  Instrument 

Instrument  development  Included  the  following  steps: 

1.  Items  were  written  for  each  of  the  75  selected  competencies. 
Three  competencies  appeared  to  contain  two  important  ideas  each;  there- 
fore, two  Items  were  written  for  each  of  these  competencies,  resulting 

4 

in  78  items  in  the  final  instrument. 

2.  The  Items  were  typed  in  the  prescribed  format  and  adminis- 
tered to  a  convenience  sample  of  three  persons  knowledgeable  in  research, 
evaluation,  and  development.   These  persons  were  asked  to  coinmcnt  on  ambiguities 
in  items  or  directions,  difficulties  with  format,  etc;  their  feedback 
resulted  in  some  revisions. 

3.  The  revised  Instrument  was  sent  in  a  second  tryout  to  a 
random  sample  of  RDDE  personnel  who  had  served  as  interviewees  In  the 


During  Item  construction,  an  attempt  was  made  to  state  the 
competencies  in  behavioral  terms.    Some  of  those  who  had  conducted  the 
interviews  reported  In  Chapter  II  felt,  however,  that  this  attempt 
resulted  in  a  distortion  of  the  Initial  Interview  data.   The  Task  Force 
staff  felt  that  It  was  not  as  important  to  state  the  Items  more  behavioral ly 
as  it  was  to  retain  the  accuracy  of  the  competency  data  obtained  in  the 
actual  Interviews. 
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collection  of  data  for  Objective  1.   Returns  from  20  fiersons  were 
reviewed  and  no  further  revisions  in  the  instrument  were  found  to  be 
necessary. 

The  final  instrument  that  resulted  from  these  steps  appears 
in  Appendix  G. 

Use  of  the  Instrument 

The  instrument  described  herein  might  be  used  in  a  variety  of 
ways.    For  example,  persons  conducting  either  pre-  or  in-service  training 
programs  in  inquiry-oriented  areas  might  use  relevant  portions  of  the 
instrument  to  measure  program  effectiveness  by  assessing  trainee 
perceptions  of  their  competencies  at  the  beginning  and/or  termination 
of  the  training.    Research-Straining  and  research-producing  agencies  might 
also  use  the  measure  to  determine  areas  where  more  training  should  be 
offered  to  employees,  trainees,  or  members  of  the  association  (e.g., 
AERA).   The  instrument  could  also  be  used  by  RDDE  personnel  or  trainees 
as  a  guide  to  areas  in  which  they  might  need  additional  training.  In 
all  the  Instances  above,  the  results  might  be  more  tenuous  than  if  mastery 
tests  of  the  type  described  in  the  follovring  chapter  were  used; 
however,  as  indicated  before,  there  are  several  reasons  why  use  of  the 
present  instrument  ma^  be  more  feasible  in  many  circumstances. 

Considerations  in  Using  the  Instrument 

There  are  three  major  considerations  in  using  the  instrument, 
each  of  which  is  discussed  briefly  below. 

First,  appropriate  directions  should  be  given  to  respondents. 
A  suggested  set  of  directions  is  included  in  Appendix  G*    These  direc- 
tions are  designed  for  use  wltb^ither  the  entire  instrument  or  any 


m 

Second,  It  must  be  determined  whether  to  administer  the  entire 
instrument  or  a  subset  of  items.    This  choice  should  be  dictated  by 
the  purpose  for  which  the  instrument  ^*s  being  administered  and  the 

relevance  of  each  task  category  to  tho^^e  purposes.    For  example, 
if  AERA  wished  to  determine  how  many  of  its  members  in  the  inter- 
mountain  states  would  profit  from  a  short-term,  regional  training  pro- 
gram in  RODE  management,  they  might  ask  members  in  those 
states  to  respond  to  an  instrument  composed  of  the  19  items  designed 
to  measure  competencies  in  task  category  3.    Items  for  any  task  category 
can  be  found  by  use  of  the  key  shown  in  Appendix  H. 

Third,  users  must  determine  whether  they  wish  to  collect 
supplemental  data  on  respondents;  for  example,  personal ogical  or 
demographic  data  on  respondents  might  be  collected  readily  by  the 
addition  of  relevant  items.    In  the  example  above,  AERA  might  need  to 
determine  whether  or  not  potential  trainees  would  be  willing  to  pay  a 
fee  for  the  training. 
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CHAPTER  IV 

DEVELOPMENT  OF  A  PILOT  TEST  OF  SELEQED  COMPETENCIES 
IN  EDUCATIONAL  RESEARCH,  EVALUATION,  DEVELOPMENT  AND  DIFFUSION 


115 


115 

DEVELOPMENT  OF  A  PILOT  TEST  OF  SELECTED  COMPETENCIES 
IN  EDUCATIONAL  RESEARCH,  EVALUATION,  DEVaOPMENT  AND  DIFFUSION^ 

Satisfactory  Instruments  designed  to  assess  the  conpetence  level  of 
educational  research  and  research-related  personnel  are  not  currently  avail- 
able.   Such  Instruments  are  needed  to  assess  competencies  and  deficiencies 
of  trainees  and  employees  In  educational  research,  development,  diffusion 
and  evaluation  (RDDE).   An  assessment  of  conpetencles  and  deficiencies  would 
provide  a  rational  basis  for  making  recommendations  regarding  future  train- 
ing programs  and  manpower  needs .  Task  Force  activities  in  developing  pilot 
tests  for  measuring  RDDE  competencies  are  described  In  this  chapter. 

In  developing  competency  measures  it  Is  not  the  purpose  of  the  Task 
Force  to  use  the  tests  to  assess  or  diagnose  individual  achievement.  Rather, 
the  purpose  Is  to  use  the  tests  to  Identify  whether  or  not  groups  of  RDDE 
trainees  and  workers  possess  certain  Important  competencies;  i.e.,  the  tests 
would  enable  one  to  make  statements  about  the  proficiency  of  groups  of 
Individuals  (or  even  the  profession  as  a  whole)  on  selected  content.  Test 
scores  might  subsequently  be  correlated  with  demographic  variables  collected 
on  examinees  to  provide  further  understanding  about  the  background  and 
training  of  RODE  personnel. 

Specifically,  the  Task  Force  has  continued  In  the  present  project  to 
attempt  to  Identity  the  skills  and  knowledge  which  are  In  the  shortest  supply— 
l.e. ,  those  In  which  training  is  most  needed. 

^All  of  the  activities  associated  with  designing,  developing,  cntiquing, 
revising  and  assessing  the  pilot  test  described  herein  were  conducted  either  by 
or  under  the  direction  of  Jason  Nlllman  and  Ellis  B.  Page,  with  assistance  of 
other  Task  Force  mAers  and  staff. 
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To  acconplish  this,  the  strategy  of  determining  discrepancies  between  present 
proficiency  and  required  proficiency  in  RDDE  skills  and  knowledge  was  adopted, 
hirst  efforts  to  identify  required  skills  were  reported  in  Technical  Papers  No. 
]_t  2,  3  and  5  in  the  Task  Force  technical  paper  series.   Objective  1  of  the 
current  Task  Force  project  represents  a  further  effort  to  identify  required 
proficiencies;  interviews  were  conducted  with  persons  carrying  out  exemplary 
RDDE  activities  to  identify  those  competencies  required  in  such  work.  (Skills 
and  knowledge  identified  through  these  interviews  are  reported  in 
Chapter  II.)     It  was  originally  intended  to  use  these  data  as  a  basis  for 
determining  what  competencies  should  be  measured  in  the  pilot  tests  described 
herein.   However,  because  of  a  USOE  request  to  complete  the  development  of  the 
pilot  tests  In  one-half  of  the  time  originally  proposed,  it  was  necessary  to 
conplete  the  test  development  at  approximately  the  same  time  Objective  1  data 
were  being  analyzed.  Consequently,  it  '.:v\s  necessary  to  use  content  from  Techni - 
cal  Papers  1  and  5  as  the  basis  for  determining  important  competencies  to  be 
included  in  the  pilot  tests.   This  has  resulted  in  the  competencies  paralleling 
closely  the  a  priori  categories  of  RDDE  rather  than  the  empirically  derived 
categories  reported  in  Chapter  II. 

The  knowledge  derived  from  administering  test  items  designed  to  measure 
present  levels  of  proficiency  should  permit  one  to  make  intelligent  statements 
about  areas  in  which  future  training  is  necessary.   A(birinisterino  the  tests  to 
sufficient  numbers  of  RDDE  persons  should  provide  a  good  profile  nf  present 
RDDE  proficiency  and  lead  to  informed  reconmendatioris  about  future  directions 
of  RDDE  training. 
Test  Developwent  Objectives 

In  the  context  described  above,  the  spcrific  t«7St  development  objective 
of  the  Task  Force  was  statt-d  as  follow.;  "lo  desiqn  and  develop  mastery  tests 
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of  certain  competencies  In  RDDE"  (Objective  2.1).    During  contract  re- 
negotiations with  OEt  that  objective  was  modified  because  of  the  shortened 
period  of  time  ailowed  for  Its  attainment.    The  final  commitment  was  to 
deliver  to  OE:  "(1)  one  completed  pilot  mastery  test  of  certain  competencies 
In  RDDE  and  (2)  a  description  of  the  test  on  all  relevant  characteristics." 
This  commitment  was  fulfilled  by  the  development  of  two  tests  shown  In  the 
appendices  and  by  the  description  of  the  tests  contained  In  this  chapter. 

The  tests  were  designed  expressly  as  pilot  tests  to  serve  as  a  basis 
for  preliminary  data  collection  and  subsequent  development  of  more  refined 
tests  of  RODE  competencies.    Consequently,  no  norming  was  undertaken,  nor 
were  final  checks  on  validity  and  reliability  made  (preliminary  Information 
concerning  validity  and  reliability  of  items  is  contained  herein). 

The  remainder  of  this  chapter  contains  (a)  a  discussion  of  test  develop- 
ment procedures,  (b)  a  description  of  the  resulting  pilot  tests,  (c)  copies 
of  the  tests,  and  (d)  a  listing  of  the  specific  skill  or  knowledge  measured 
In  each  item. 

Test  Development  Procedures 

This  section  Includes  a  discussion  of  (a)  specification  of  test 
content,  (b)  attempts  to  identify  existing  items  that  might  be  u!;e<1,  (c) 
item  construction  procedures,  and  (d)  item  critique,  tryout,  and  revision 
procedures. 

Specification  of  Test  Content 

Ideally,  a  conplete,  empirically  based,  empirical  map  of  RDDE,  includ- 
ing listing  of  1iifX)rtant  skills  and  knowledge  would  have  been  available  as  the 
basis  for  determining  the  content  of  the  pilot  test.   However,  as  mentioned 


118 


118 


earlier,  the  speed  up  of  test  development  required  that  test  content  be 

chosen  prior  to  the  completion  of  the  objective  1  data  analysis  reported 

in  Chapter  II;  consequently,  earlier  a  priori  lists  of  essential  skills 

and  knowledge  were  used  to  determine  content.   Within  the  parameters 

of  these  lists,  it  was  necessary  to  choose  between  (a)  attempting 

to  test  all  or  most  of  the  specific  skills  and  knowledge  viewed 

as  important,  or  (b)  testing  a  smaller  number  of  the  most  important  skills  or 

knowledge.    The  Task  Force  selected  the  latter  option— a  sampling  of  important 
competencies  in  the  RODE  domain.    It  was  felt  that  this  was  the  more  defensible 
choice,  for  two  reasons.    First,  it  was  much  more  feasible  than  attempting  to 
measure  all  the  skills  and  knowledge  viewed  as  important;  budget  and  time  re- 
strictions precluded  a  comprehensive  test  of  all  RDDE  skills  even  if  they  could 
have  been  identified.   Second,  working  with  selected  competencies  would  still 
allow  some  valid  inferences  to  be  drawn  even  beyond  the  actual  competencies 
measured,  since  there  are  substantial  inter-relationships  among  many  competen- 
cies. 

The  competencies  used  as  a  basis  for  writing  the  test  items  were  chosen 
by  use  of  the  following  criteria: 

(1)  Is  the  competency  trainable,  i.e.,  does  it  include  knowledge  or  skill 
which  can  be  communicated     through  systematic  training  (as  opposed  to  personal 
characteristics  which  are  unlikely  to  be  enhanced  through  training)? 

(2)  Is  the  competency  the  end  product  of  substantive  training  in  a 
field  outside  of  educational  RDDE  (i«e.,  should  the  competency  be  gained  else- 
where)? 

(3)  Is  the  competency  important  or  significant  (as  opposed  to  trivial)? 

(4)  Is  the  competency  likely  to  be  useful  in  many  relevant  activities 
(i.e.,  is  it  commonly  required  in  a  broad  range  of  rdde  activities)? 

(5)  Can  the  competency  be  efficiently  tested  in  a  pilot  test  of  the 
type  proposed  by  the  Task  Force?  119  ^  
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Application  of  these  criteria  resulted  in  selection  of  both  skills 

and  knowledge  in  the  final  list  of  competencies  to  be  tested.   There  were 

approximately  twice  as  many  "knowledge  competencies"  as  "skill  competencies" 

chosen.    Direct  assessment  of  performance  (i.e.,  use  of  skills)  was  viewed 

dS  requiring  items  which  would  be  time  consuming  to  administer,  resulting  in 

a  narrower  sampling  of  important  content*    Knowledge  was  viewed  as  necessary 

to  be  effective  in  RDDE  and  was  thought  in  many  cases  to  be  a  necessary 

precursor  to  skill.    Assessment  of  knowledge  was  thought  to  provide  more 

efficient  Items  and,  therefore,  enable  a  broader  sampling  of  relevant  content. 
A  second  decision  relating  to  content  was  the  relative  emphasis  which 

should  be  placed  on  RDDE.    Three  decisions  were  reached  and  are  discussed 

below: 

(1)  Diffusion  skills  and  knowledge  would  not  be  included  In  the  test. 
This  decision  was  reached  for  two  reasons.    First,  none  of  the  Task  Force 
ment)ers  or  staff  had  particular  expertise  in  this  area  and  it  seemed  audacious 
In  the  extreme  to  attempt  to  write  items  in  an  area  where  the  writers  were  not 
thoroughly  prepared. 

(2)  Development  skills  and  knowledge  would  be  included  in  the  test. 
This  decision    was  made  with  the  provision  that  it  be  contingent  upon  the 
ability  of  W.  James  Popham,  the  Task  Force  member  viewed  as  having  expertise 
In  educational  development,  to  identify  critical  development  knowledge  and 
skills  and  write  items  to  assess  these  competencies. 

(3)  Research  and  evaluation  skills  and  knowledge  would  be  included  in 
the  test.    The  expertise  of  most  Task  Force  members  (including  the  present 
authors)  Is^y  in  these  areas  and  it  was  felt  that  the  Task  Force's  major  contri- 
bution In  this  pilot  test  ^^y  in  devising  items  to  assess  competencies  in  these 
areas.   Using  the  criteria  discussed  earlier,  research  and  evaluation  competen- 
cies to  be  assessed  were  drawn  from  previous  Task  Force  lists  (see  Technical 


Paper  No.  5)  and  included  the  following: 
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Research  Skills 

Rl.    Drawing  implications  from  results  of  prior  research. 

R2.    Formulating  hypotheses  or  questions  to  be  answered  by  the  research. 

R3.    Identifying  the  population  to  which  results  should  be  generalized  and 
selecting  a  sample  of  the  population. 

R4.    Applying  experimental  design  and  recognizing  and  controlling  threats  to 
validity. 

R5.    Selecting  or  developing  techniques  of  measurement. 
R6.    Assessing  the  validity  of  measurement  techniques. 

R7.    Utilizing  appropriate  data  gathering  methods  (tests,  interviews,  analysis 
of  documents,  etc.) 

R8.    Understanding  the  general  role,  types,  and  assumptions  of  statistical  tech- 
niques and  drawing  on  such  knowledge  in  using  appropriate  techniques  of 
data  analysis. 

R9.    Interpreting  and  drawing  appropriate  conclusions  from  data  analysis. 
Evaluation  Skills 

El.    Translating  broad  objectives  into  specific,  observable  objectives. 

E2.    Identifying  standards  or  norms  for  judging  worth. 

E3.    Selecting  (or  developing)  and  using  valid  techniques  of  measurement  to 
yield  information  on  outcomes. 

E4.    Employing  appropriate  techniques  of  statistical  analysis. 
Knowledge  of  Statistics 

SI.    Descriptive  techniques,  their  definition  and  interpretation,  including 
measures  of  central  tendency,  variation,  correlation  and  prediction. 

32.    Fundamental  theorems  of  finite  sample  space  theory  (nature  of  a  sample 
space  and  an  event,  addition  rule,  multiplication  rule,  etc.) 

S3.    Definitions  and  properties  of  the  principoi  continuous  (normal,  chi-square, 
t,  F)  and  discrete  (binomial,  multinomial)  probability  distributions. 
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54.  Major  schools  of  thought  on  statistical  inference  (Neyman-Pearson, 
Fisherian,  Bayesian,  likelihood  estimation),  including  principal 
concepts  thereof. 

55.  Nature  and  use  of  the  general  linear  model  including  least-squares 
estimation  and  distribution  theory,  which  includes  analysis  of  vari- 
ance methods  and  the  extension  of  comparative  experimental  analysis 
to  post  hoc  multiple  comparisons  techniques,  including  Scheffe, 
Tukey,  Dunnett  and  Dunn. 

56.  Consequences  of  failure  to  meet  assumptions  of  principal  parametric 
inferential  techniques. 


Knowledge  of  Experimental  Design 

Dl .    Library  knowledge:   names  of  major  books  and  jouvnals,  their  authors, 
some  familiarity  with  content,  and  knowledge  of  the  value  ascribed 
to  each  work  by  leaders  in  the  discipline. 

D2.    Randomization  as  a  means  of  experimental  control  and  its  relation- 
ship to  inferential  statistical  methods. 

D3.    Factors  affecting  the  internal  and  external  validity  of  experimental 
and  quasi -experimental  designs. 

D4.    Definition  of  fixed  effects,  random  effects,  and  mixed  effects  de- 
signs; crossed  and  nested  factors;  the  nature  of  interactions,  their 
graphing  and  interpretation. 

D5.    Nature  and  problems  in  the  use  and  analysis  of  repeated  measures 
designs. 
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Knowledge  of  Psychometrics 

PI.    Library  knowledge  (see  Dl  above). 

P2.    Fundamental  theorems  of  classicaV  true-score  theory  (reliability 
coefficient,  variance  error  of  measurement,  correction  for  attenua- 
tion, relationship  of  test  length  to  the  variances  of  x,  t,  e,  and 
reliability  coefficients). 
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P3.   Types  of  test  reliability  and  validity. 
P4.   Reliability  and  validity  of  sums. 

P5.    Fundamental  postulate  and  theorems  of  components  and  common-factor 
analysis; 

P6.    Four  factor  analysis  models:    components  analysis,  image  analysis, 
canonical  factor  analysis,  alpha  facto?vanalysis. 

P7.   Methods  of  factor  rotation  (orthogonal)   and  transformation  (oblique): 
varimax,  equamax,  quartimax,  promax,  Harris -Kaiser. 

P8.   Factors  affecting  the  size  of  measures  of  relationship. 

Knowledge  of  Measurement 

Ml.   Library  knowledge  (see  Dl  above). 

M2.   Definition  and  properties  of  nominal,  ordinal,  interval  and  ratio 
measurement  scales. 

M3.   Parameters  and  properties  of  the  major  test-score  scales  including 
T  scores,  z  scores,  CEEB  scores,  ratio  and  deviation  IQ  scores, 
grade  equivalent  scores,  and  percentile  scores. 

M4.   Major  forms  of  assessment  of  knowledge  and  cognitive  skills,  includ- 
ing multiple  choice,  completion,  free  response,  ranking,  matching,  etc. 

M5.   Primary  methods  of  assessing  attitudes,  including  Likert  and  Thurstone 
scales,  interests  and  social  perception,  including  semantic  differen- 
tial and  Q-sort. 

M6.   Construction  and  use  of  rating  scales,  including  methods  of  assess- 
ing rater  agreement. 

General  Knowledge 

Gl .   The  nature  of  theories,  models,  and  paradigms  in  the  social  sciences 
(including  the  philosophy  of  science). 
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G2.   General  library  knowledge  of  research  and  evaluation  (see  Dl  above). 
G3.   Knowledge  of  the  computer  and  computer  programning. 

NOTE:    From  this  point  on,   the  numbering  system  employed 
above  will  be  used  to  refer  to  the  particular  skills  and  knowledge  measured 
by  the  test.   That  is,  research  skills  will  be  noted  by  an  "R,"  evaluation 
skills  by  an  "E,"  statistics  knowledge  by  an  "S,"  etc.    It  should  also  be 
pointed  out  that  all  the  knowledge  items:   statistics  (S),  design  (D), 
psychometrics  (P) ,  measurement  (M),  and  general  (G)  are  grouped  together 
under  the  heading  research  and  evaluation  (R-E)  knowledge. 

Search  for  Existi ng  I terns 

An  attempt  was  made  to  determine  if  there  were  already  existing 
items  which  could  be  adopted  or  adapted  for  use  in  the  pilot  test.  Con- 
tacts were  made  with  the  Educational  Testing  Service  (ETS)  and  the  National 
Council  on  Measurement  in  Education  (NCME)  to  see  if  there  were  pools  of 
items  or  tests  relating  to  RDDE  content  which  might  be  useful  for  such 
purposes. 

ETS  items.   The  following  ETS  tests  were  examined  for  possible  items: 

1.  Graduate  Record  Examinations  (Psychology,  Education,  Sociology, 
and  Philosophy) 

2.  Undergraduate  Record  Examinations  (Psychology,  Education,  and  . 
Sociology) 

3.  College  Level  Examination  Program  (Computers  and  Data  Processing, 
Educational  Psycliology,  General  Psychology,  History  of  American 
Education,  Human  Growth  and  Development,  Statistics,  Tests  and 
Measurements,  and  the  pretest  on  Elementary  Computer  Programming.) 
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4.  National  Teacher  Examinations  (Conmon  Exams— Professional 
Education;  Early  Childhood  Education;  Education  in  the  Elemen- 
tary School;  and  Reading  Special ist--Elementary  School) 

5.  Teacher  Education  Examination  Program  (General  Professional 
Exams,  Early  Childhood  Education,  Elementary  School  Education). 

There  were  found  to  be  few  items  in  any  of  the  above  tests  which 

were  relevant  for  the  specific  purposes  of  this  activity.    It  was  felt  that 

the  gain  in  using  the  few  items  found  to  be  relevant  was  not  worth  the  losses 

associated  with:    (a)    the  efforts  necessary  to  gain  permission  to  use  the 

items,  and  (b)  the  fact  that  particular  items  selected  for  relevance  to 

RDDE  would  not  represent  a  sample  of  any  general izable  sort  and  therefore 

could  not  be  related  meaningfully  to  ETS  norms  on  the  respective  tests. 

Consequently,  no  ETS  items  were  used  in  the  pilot  test  described  herein. 

However,  some  items  in  the  ETS  tests  did  suggest  formats  or  approaches 

to  assessment  which  were  adopted  in  writing  some  of  the  items  for  the 

pilot  test. 

NOME  Items.   NOME'S  Competency  Tests  in  Tests  and  Measurement 
(Forms  A  and  B)  were  also  examined.    Although  some  of  the  items  suggested 
item  formats  which  were  used  in  writing  some  pilot  test  items,  none  of 
the  NCME  items  were  used  in  the  present  pilot  tests. 

Item  Construction 

A  pool  of  230  original  items  on  research  and  evaluation  was 
written  by  Drs.  Millman  and  Page.   A  pool  of  23  development  items  was 
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submitted  by  Dr.  Popham.     Both  pools  of  Items  were  almost  exclusively 
objective  Items,  reflecting  an  earlier  decision  to  adhere  to  this  type  of 
item  In  the  pilot  test.   This  decision  resulted  from  early  critiques  of 
preliminary  items  relating  to  time  consumption  and  feasibility.  Although 
It  was  acknowledged  that  there  Is  Inevitably  some  hiatus  between  objective, 
paper  and  pencil  testing  and  "live  performance"  (and  perhaps  more  so 
than  with  more  "naturalistic"  performance  tests),  other  considerations 
inlluenced  the  decision.    For  example,  evaluating  direct  performance  items 
throughout  tryout  phases  requires  greater  expenditures  of  time  and  money 
than  were  available  for  the  development  of  this  pilot  test.  Secondly, 
objective  tests  allowed  a  greater  number  of  competencies  to  be  assessed 
efficiently.   Third  (and  perhaps  most  important),  there  were  serious 
concerns  about  the  test  being  used  by  persons  with  limited  expertise  in 
the  content  being  measured.   To  the  extent  this  occurred,  it  seemed 
critical  to  provide  an  objective  scoring  format  rather  than  perpetuating 
the  fallacy  that  untrained  persons  could  score  essay-type  responses 
validly.    Finally,  it  seemed  that  a  majority  of  the  Important  competen- 
cies could  be  tested  as  readily  with  objective  as  with  other  types  of 
items. 


These  items  were  drawn  primarily  from  items  Popham  had  written 
earlier,  some  of  which  had  also  been  used  by  the  author's  permission  in 
"Evaluation:   The  Development  of  Instructional  Products,  Form  X."  South- 
west Regional  Educational  Laboratory,  Inglewood,  California,  1967. 
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Item  Critique,  Tryout,  and  Revision 

For  the  research  and  evaluation  items,  the  critique  included  the 
following  steps:    (1)  Drs.  Nillman  and  Page  critiqued  one  another's  Items, 
(2)  all  items  were  critiqued  in  terms  of  content  and  technical  adequacy 
by  two  expertfi  in  research  and  evaluation,   and  (3)  Nillman  and  Page  revised 
the  items  on  the  basis  of  the  three  independent  critiques  of  each  item. 

The  development  items  were  critiqued  by  Nillman  and  Page  In  terms 

of  general  technical  considerations  (not  content).   Those  items  viewed 

as  adequate  on  technical  grounds  were  critiqued  by  an  expert  in  educational 
4 

development,  who  was  asked  to  do  the  following: 

"1.   Assess  whether  the  items  measure  important  competencies  (skills 
or  knowledge  In  educational  development.   Note  any  items  which 
you  feel  do  not  measure  important  development  competencies. 

2.  Assess  whether  the  set  of  items  falls  to  include  some  important 
development  competencies.  .  .  .  If  so,  could  you  write  a  few 
items  to  cover  such  additional  important  competencies? 

3.  Assess  whether  the  Items  are  good  items  technically.    If  not, 
could  you  suggest  modifications  to  improve  them?"  (personal 
correspondence  to  Leslie  Briggs,  July  28,  1971  •) 

In  the  response  to  the  above  points,  (1)  it  was  indicated  in  the  critique  that 
all  of  the  original  items  were  relevant  to  the  development  process,  (2)  it  was 
pointed  out  that  the  original  Items  were  exclusively  on  the  developiriant  of  in- 
structional products  and,  consequently,  additional  items  of  broader  scope  were 
suggested,  and  (3)  revisions  were  suggested  in  some  items  because  of  technical 
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Dr.  Gene  V  Glass  and  Dr.  Kenneth  D.  Hopkins. 
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Dr.  Leslie  J.  Briggs. 
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inadequacies.   Thirty-three  additional  items  were  submitted  by  Briggs,  16 
of  which  were  discarded  because  of  technical  inadequacies.   The  resulting  item 
pool  was  thus  increased  (by  the  addition  of  the  17  remaining  items),  but 
still  emphasized  primarily  development  of  instructional  products. 

After  revision  of  all  items,  including  discarding  "bad"  items,  there 
were  137  research  and  evaluation  items  and  40  development  items  in  the  item 
pool.   After  this  initial  screening,  it  was  intended  that  all  items  were 
to  be  ".  .  .  tried  out  with  a  small  nunter  of  subjects  to  identify  antigui- 
ties  in  items,  derive  preliminary  estimates  of  item  difficulty,  etc.  This 
will  not  represent  an  attempt  at  norming,  however.  .  ."  (amendment  to  USOE 
grant  0EG-O-71-0617(52O)).  This  tryout  did  not  represent  an  attempt  at 
norming  (even  on  a  pilot  basis);  that  and  routine  reliability  and  validity 
assessment  were  to  be  conducted  subsequently.    Only  preliminary  assessment 
in  these  areas  was  viewed  as  appropriate  to  the  development  of  the  pilot 
tests.  <i- 

In  choosing  the  tryout  group,  an  attempt  was  made  to  administer  each 
pool  of  items  to  persons  known  to  have  relevant  knowledge  and  skills.  This 
resulted  in  a  convenience  sample  of  faculty  members  and  graduate  students 
in  six  major  universities.  Each  member  of  the  sample  was  chosen  on  the 
basis  of  being  skilled  or  knowledgeable  (to  at  least  some  degree)  about 
the  relevant  content— i  .e.,  there  were  no  persons  who  were  viewed  as  com- 
pletely naive  in  the  content  area.^  On  this  basis,  100  persons  were 

5 

The  selection  was  made  on  the  basis  of  recommendations  of  either 
Task  Force  members  or  colleagues  who  were  thoroughly  instructed  in  the  types 
of  individual  desired  for  inclusion  in  the  tryout.  The  following  persons 
nominated  persons  for  inclusion  at  the  following  institutions:   Ronald  D. 
Anderson  and  Blaine  R.  Worthen  (University  of  Colorado);  Leslie  J.  Briggs 
(Flonds  State  University);  Tom  Hastings  (University  of  Illinois);  Ellis  B. 
Page  {University  of  Connecticut);  W.  James  Popham  (University  of  California 
at  Los  Angeles);  and  James  R.  Sanders  (Indiana  University). 
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Identified  and  asked  to  assist  with  the  test  development.  Seventy-six 
persons  compiled  with  this  request.  The  two  item  pools  (development 
items  in  one  and  research  and  evaluation  items  in  the  other)  were  repro- 
duced in  two  separate  "tests"  for  the  tryout.   The  nunter  and  type  of 
persons  who  took  each  of  these  tests  are  shown  in  Table  15. 


Table  15 

Type  and  Locale  of  Persons  Participating  in  ItemTryout 


Test 

Institution 

Number  of  Tests : 
Sent  Returned 

Persons  Completing  Tests: 
Faculty     Graduate  Students 

U.  of  Connecticut 

15 

11 

4 

7 

Research  and 

U.  of  Illinois 

10 

8 

4 

4 

Evaluation 

Indiana  U. 

10 

7 

5 

2 

U.  of  Colorado 

10 

9 

1 

8 

uaA 

25 

17 

0 

17 

Development 

Florida  State  U. 

15 

11 

3 

8 

U.  of  Colorado 

15 

13« 

12  • 

1 

Totals: 

100 

76 

29 

47 

Five  of  these  persons  did  not  complete  all  items;  however,  they  wrote  a  critique  of 
the  test  itself. 

The  76  persons  who  responded  included  faculty  at  all  academic  ranks, 
doctoral  students,  and  a  few  advanced  master's  degree  candidates.   With  few 
exceptions,  these  were  persons  with  demonstrated  competence  in  the  relevant 
areas.    For  example,  the  graduate  students  and  faculty  who  took  the  research 
and  evaluation  test  were  drawn  primarily  from  the  following  programs: 
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educational  research  and  psychology,  the  University  of  Connecticut;  Center 
for  Instructional  Research  and  Curriculum  Evaluation,  the  University  of 
Illinois;  educational  psychology,  Indiana  University;  and  the  Laboratory 
of  Educational  Research,  University  of  Colorado.    For  the  development 
test,  examinees  were  graduate  students  from  the  Instructional  Product 
Development  program,  UCLA;  faculty  and  graduate  students  from  the  educa- 
tional development  program,  Florida  State  University;  and  faculty  members 
(or  full-time  professional  staff)  of  three  curriculum  development  projects 
at  the  University  of  Colorado  (Biological  Sciences  Curriculum  Study y  Earth 
Sciences  Curriculum  Project,  and  the  Social  Science  Educational  Consortium). 

Examinees  were  asked  to:   (1)  attempt  to  respond  correctly  to  each 
item  on  the  test  administered  to  them,  and  (2)  write  any  comments  they 
had  relating  to  inadequacies  or  anrtbiguities  in  the  items. 

Most  comments  were  directed  at  item  ambiguities,  unintended  cues  to 
the  correct  response,  etc.,  and  were  used  in  revising  the  items.  However, 
on  the  development  test,  there  were  several  negative  comments  relating  to 
whether  or  not  the  test  measured  important  development  competencies.  Five 
persons  from  the  Biological  Sciences  Curriculum  Study  (BSCS)  stopped  short 
of  responding  to  all  development  items  because  of  their  feeling  that  these 
represented  a  superficial  and  misdirected  approach  to  curriculum  develop- 
ment.  Their  concerns  are  reflected  in  the  following  statement: 
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This  .  .  .  effort  [the  Task  Force  attempt  to  measure 
educational  development]  1s»  first  of  a11»  focused  on  the 
wrong  target,  namely  a  focus  on  people  rather  than  on  their 
products.   Studies  of  creativity  and  creative  processes 
were  apparently  ignored  resulting  In  a  naive  view  of  curri- 
culum development  processes  and  competencies.    If  one  were 
to  center  research  on  competencies  of  development  personnel , 
creativity  would  be  an  Important  characteristic.^  Note 
that  the  very  statement  of  the  AERA  Task  Force  Is  ndlve, 
centering  on  the  measurement  of  knowledge  and  skills. 
(Personal  communication  from  Jim  Robinson,  Richard  Tolman, 
Jim  Eckenrod,  Fred  Riaismussen,  and  Joe  Steele,  August  ^4, 
1971.    Footnote  reference  added.) 


Since  these  reactions  suggested  that  a  completely  different  approach 
to  measuring  development  competencies  should  be  taken,  they  were  not  used 
In  revising  the  pilot  test  In  development.    They  are  reported  here,  however, 
to  alert  the  reader  to  the  fact  that  there  Is  a  strong  minority  view  which 


Research  of  the  Task  Force  on  competencies  of  development  personnel 
(reported  in  Chapter  II)   lends  some  support  to  this  statement. 
Several  competencies  had  some  relationship  to  "creativity."    It  should  be 
noted  here,  however,  that  the  Task  Force  has  no  Intent  to  overlook  creati- 
vity.   Indeed,  several  discussions  centered  around  the  Importance  of  creati- 
vity and  several  affective  variables  in  educational  development  before 
It  was  decided  that  skills  and  knowledge  were  the-  competencies  which  could 
be  measured  at  this  time.   Albeit  an  Important  characteristic,  creativity 
Is  without  reputable  means  of  measurement  and  was  omitted  for  this  reason 
and  not  because  of  any  nalvet^  about  how  important  a  role  It  plays. 
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suggests  that  the  concept  of  development  reflected  in  this  pilot  test  is 
rejected  by  some  prominent  curriculum  developers.^ 


This  reaction  of  the  BSCS  examinees  simply  highlights  the  fact  that 
there  is,  at  this  point  in  time,  no  real  consensus  as  to  what  competencies  are 
important  in  educational  development;  indeed,  "developers"  are  not  of  one  mind 
concerning  what  development  comprises.   There  seem  to  be  at  least  two  distinct 
schools  of  thought  about  development  of  instructional  products.    One  view  (that 
implicit  in  the  work  of  the  Task  Force  and  outside  expert  who  developed  the 
development  items)  has. wide  currency  among  educationists  who  have  established 
programs  to  train  persons  in  product  development  (e.g.,  the  UCLA  Instructional 
Product  Development  Program,  the  educational  development  programs  at  the  Florida 
State  University  and  the  Pittsburgh  R  and  D  Center,  and  the  Department  of  In- 
structional Research  and  Development  at  Brigham  Young  University),  stating 
measurable  objectives,        identifying  entry  capabilities  of  learners,  sequen- 
cing concepts  in  instruction,  and  conducting  formal  evaluations  of  the  finished 
products  are  stressed  in  this  approach  as  essential  adjuncts  to  the  actual 
writing  of  the  materials.   The  second  view  (that* expressed  by  the  BSCS  exami- 
nees) seems  to  be  most  predominant  among  persons  engaged  in  contract  development 
of  currtculum  materials  or  packages— often  scholars  from  the  various  disciplines 
(e.g.,  BSCS,  PSSC,   ICSM).   Here  the  approach  to  development  seems  to  be  to 
gather  a  team  of  bright,  creative  individuals  with  knowledge  about  the  relevant 
disciplines  and  focus  the  production  of  the  materials  around  how  to  present  in 
an  effective,  interesting  fashion  the  basic  content  and  structure  of  the  disci- 
pline.   Persons  in  this  second  school  of  thought  seem  to  view  adherents  to  the 
first  as  proponents  of  misguided,  mechanistic  approaches  to  development  which 
will  result  in  uninteresting,  sterile  products.    Proponents  of  the  first  view 
are  likely  to  view  their  counterparts  as  engaging  in  unsystematic  development 
efforts  which  are  completed  with  too  little  real  evaluation  or  specification  of 
precise  behaviors  or  concepts  which  are  to  be  taught.   There  are  also  differ- 
ences relating  to  the  issue  of  "who  is  the  developer?"   Educationists  conducting 
training  programs  seem  to  assume  that  one  provides  important  development  con- 
cepts and  skills  to  each  trainee  (e.g.,  skills  in  sequencing,  evaluation,  and 
using  A-V  equipment)  and  that  most  trainees  go  on  to  be  development  generalists-- 
i.e.,  you  train  educational  developers,  £er  se.    Developers  on  curriculum  develop- 
ment projects  seem  to  view  development  as  a  team  effort  and  view  with  suspicion 
persons  trained  "superficially"  in  a  broad  sampling  of  development  or  develop- 
ment-related competencies.  They  prefer  to  recruit  several  persons,  each  of  whom 
has  specialized  training  relevant  to  the  project  (e.g.,  an  evaluator,  an  A-V 
man,  several  persons  trained  in  the  relevant  subject  matter,  an  editor  or  journa- 
list).   In  their  view  you  do  not  train  educational  developers  per  se,  but  make 
up  development  teams  of  bright,  creative  persons  with  expertise  gained  in  rele- 
vant, established  training  programs. 


132 


^       u/  event,  there  are  at  least  two  schools  of  thought 

regarding  the  proper—or  most  fruitful— approach  to  the  educational  develop- 
ment process  and,  consequently,  two  very  different  views  of  the  assessment 
of  development  competencies.    It  seems  clear  that  there  can  be  no  single 
acceptable  approach  to  measuring  these  competencies  until  there  is  substan- 
tial agreement  on  what  constitutes  the  educational  development  process 
itself.     Although  both  viev/s  described  above  have  doubtlessly  been  over- 
simplified (if  not  caricatured),  their  existence  points  to  a  serious  diffi- 
culty which  Is  likely  to  be  encountered  In  any  effort  to  define  educational 
development  either  through  measurement  efforts  such  as  that  described  herein 
or  the  establishment  of  training  programs  by  funding  agencies.   This  is  an 
interesting  but  serious  problem  confronting  the  USOE  in  their  efforts  to 
establish  programs  to  train  educational  developers.    It  may  be  that  attempt- 
ing to  identify  the  critical  content  of  development  by  bringing  together 
adherents  of  both  schools  of  thought  (e.g.,  having  BSCS  personnel  sit  down 
with  persons  like  Popham  and  Briggs)  in  a  small  conference  to  react  to 
data  on  development  competencies  reported  in  Chapter  II  would  be  a  useful 
precursor  to  further  plans  to  establish  programs  to  train  educational 
developers.   Such  efforts  ma^y  at  least  partially  eliminate  the  present 
disagreements  about  development  which  are  uncomfortably  akin  to  the  fabled 
attempts  of  the  blind  men  to  describe  the  elephant. 
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Examinee  responses  to  all  items  were  scored,  coded,  keypunched,  and 
submitted  to  a  computer  item  analysis  (Laboratory  of  Educational  Research 
FORTAP  program).   Item  analyses  were  run  separately  cn  the  40  development 
items  and  the  137  research  and  evaluation  items.   The  resuUs  of  these 
analyses,  including:    (1)  a  reliability  estimate,  (2)  a  mean  score  for 
all  examinees,  (3)  measures  of  variability,  and  (4)  for  each  item,  the 
difficulty  level,  point-bi serial  r,  and  number  responding  to  each  option, 
are  presented  in  Tables  16  and  17. 


Table  16 


Summary  of  Item  Analysis  of  Research  and  Evaluation  Items 

(N  =  35) 


Item 

Difficulty 

Point 

Number  Choosing  Each  Option 

No. 

Level 

Bi serial  r 

A       B        C       D  E 

1  .97 

2  .83 

3  .89 

4  .91 

5  .60 

6  .89 

7  .94 

8  .89 

9  .37 

10  .89 

11  .94 

12  .77 

13  .66 

14  .57 

♦correct  response 


.471 

0 

0 

34* 

1 

0 

.002 

0 

29* 

6 

0 

0 

.080 

31* 

2 

0 

0 

1 

-.019 

32* 

3 

0 

0 

0 

.184 

21* 

13 

0 

0 

0 

.130 

4 

31* 

0 

0 

0 

-.107 

33* 

2 

0 

0 

0 

.422 

31* 

4 

0 

0 

0 

.094 

9 

6 

2 

13* 

5 

.108 

0 

3 

31* 

1 

0 

.020 

0 

33* 

2 

0 

0 

.272 

1 

6 

27* 

1 

0 

.218 

0 

0 

2 

10 

23* 

.110 

0 

7 

3 

5 

20* 

Table  16,  continued 


Item  Difficulty  Point  Number  Choosing  Each  Option 

No.  Level  Biserial  r  A       B       C       D  E 


15 

.83 

.411 

0 

4 

29* 

0 

16 

1.00 

.000 

0 

35* 

0 

0 

0 

17 

.97 

-.020 

1 

34* 

0 

n 
\t 

0 

18 

.86 

.011 

5 

30* 

n 
\t 

0 

0 

19 

.36 

.277 

5 

30* 

0 

n 
\t 

0 

20 

.63 

.235 

22* 

12 

n 
\t 

n 

u 

n 

21 

.94 

.480 

1 

0 

33* 

1 

1 

0 

22 

.60 

.174 

14 

21* 

0 

n 

0 

23 

.57 

.546 

20* 

0 

3 

3 

24 

.80 

.198 

28* 

6 

Q 

0 

0 

25 

.71 

.459 

0 

1 

25* 

7 

0 

26 

.49 

.181 

17* 

n 
\t 

n 

13 

A 
■t 

27 

.43 

.435 

15* 

3 

n 

u 

1 

16 

28 

.54 

.386 

16 

19* 

0 

0 

0 

29 

.  U"t/ 

A 

n 
U 

n 
U 

n 
U 

30 

.94 

.011 

1 

1 

33* 

0 

0 

31 

.69 

.210 

1 

24* 

0 

7 

3 

32 

.97 

-.211 

0 

34* 

1 

0 

0 

33 

.91 

.216 

2 

0 

1 

32* 

0 

34 

.60 

.369 

4 

6 

21* 

1 

1 

35 

.94 

-.097 

0 

33* 

0 

1 

0 

36 

.77 

.391 

7 

27* 

0 

0 

0 

37 

.57 

.35JJ 

20* 

2 

1 

12 

0 

38 

.60 

.086 

3 

4 

21* 

7 

0 

39 

.57 

.395 

20* 

10 

4 

1 

0 

40 

.63 

.296 

3 

2 

8 

22* 

0 

41 

.94 

.030 

1 

33* 

1 

0 

0 

42 

.86 

.439 

2 

0 

1 

2 

30* 

♦correct  response 
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Table  16,  continued 
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Item 
No.' 


Difficulty 
Level 


Point  Number  Choosing  Each  Option 

Biserial  r  A       B       C       D  E 


43 

.77 

.153 

7 

27* 

0 

0 

1 

44 

.54 

.099 

19* 

0 

1 1 

0 

5 

45 

.63 

.273 

22* 

1 

8' 

2 

0 

3 

46 

.97 

.471 

1 

34' 

Ir 

0 

0 

0 

47 

.94 

-.175 

33* 

2 

0 

0 

0 

48 

.86 

.258 

3 

2 

0 

30* 

0 

49 

.66 

.447 

23* 

5 

1 

0 

7 

0 

50 

.71 

.600 

1 

25*i 

0 

9 

0 

51 

.71 

.213 

3 

4 

25* 

3 

0 

52 

.66 

.266 

0 

5 

23* 

1 

6 

53 

.97 

-.211 

0 

1 

34* 

0 

0 

54 

.91 

-.165 

0 

32* 

3 

0 

0 

55 

1.00 

.000 

0 

0 

n 

u 

56 

.94 

-.001 

33* 

1 

1 

0 

0 

57 

.57 

.267 

1 

20* 

14 

0 

0 

58 

.31 

.206 

11* 

7 

16 

0 

0 

59 

.66 

.567 

4 

23* 

7 

0 

0 

60 

.97 

-.020 

34* 

1 

0 

0 

0 

61 

.91 

-.060 

3 

32* 

0 

0 

0 

62 

.91 

.159 

0 

32* 

1 

1 

1 

63 

.86 

.180 

0 

0 

30* 

0 

5 

64 

.89 

.251 

1 

31* 

0 

3 

0 

65 

.49 

.253 

17* 

17 

0 

0 

1 

66 

.31 

.411 

11* 

2 

0 

2 

19 

67 

.66 

.203 

23* 

11 

0 

1 

0 

68 

.77 

.342 

8 

27* 

0 

0 

0 

69 

.77 

.348 

8 

27* 

0 

0 

0 

♦correct  response 
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Table  is,  continued 


Item  Difficulty  Point  Number  Choosing  Each  Option 

No.  Level  Biserial  r  A       B       C       D  E 


70 

.74 

.327 

26* 

9 

0 

0 

0 

71 

.80 

.073 

28* 

7 

0 

0 

0 

72 

.77 

.240 

27* 

0 

0 

0 

7 

73 

.71 

.625 

0 

5 

25* 

3 

2 

74 

e  -.77 

.559 

0 

0 

1 

7 

27* 

75 

.94 

.079 

33* 

2 

0 

0 

0 

76 

.83 

.399 

29* 

5 

0 

0 

0 

77 

.63 

.230 

13 

22* 

0 

0 

0 

78 

.80 

.368 

7 

28* 

0 

0 

0 

79 

.89 

.101 

31* 

3 

0 

1 

0 

80 

.91 

.232 

1 

0 

0 

32* 

0 

81 

.97 

.471 

0 

0 

34* 

1 

0 

82 

.94 

.  .020 

1 

1 

0 

33* 

0 

83 

.97 

.062 

0 

0 

34* 

1 

0 

84 

1.00 

.000 

0 

0 

35* 

0 

0 

85 

.97 

-.020 

34* 

0 

1 

0 

0 

86 

.94 

.353 

0 

33* 

2 

0 

0 

87 

.80 

.464 

0 

6 

28* 

1 

0 

88 

.66 

.337 

7 

23* 

4 

0 

0 

89 

.89 

-.077 

3 

0 

1 

31* 

0 

90 

.60 

.355 

0 

21* 

4 

8 

0 

91 

.66 

.323 

10 

1 

23* 

1 

0 

92 

.83 

.465 

5 

29* 

1 

0 

0 

93 

.51 

.519 

5 

4 

8 

18* 

0 

94 

.91 

.346 

2 

32* 

0 

1 

0 

95 

.97 

-.292 

0 

34* 

0 

0 

0 

96 

.91 

-.052 

32* 

0 

0 

2 

0 

97 

.34 

.160 

2 

12* 

6 

0 

14 

♦correct  response 
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Table  16,  continued 


Item  Difficulty  Point  Number  Choosing  Each  Option 

No.  Level  Biserial  r  A       B       C       D  E 


98 

.83 

.092 

0 

3 

29* 

2 

0 

99 

.77 

.413 

27* 

1 

3 

2 

1 

100 

.74 

.166 

4 

3 

26* 

2 

0 

101. 

.63 

.151 

22* 

12 

1 

0 

0 

102 

.43 

-.235 

0 

18 

15* 

2 

0 

103 

.71 

-.138 

4 

25* 

6 

0 

0 

104 

.29 

.174 

10* 

13 

12 

0 

0 

105 

.80 

.283 

28* 

5 

1 

0 

0 

106 

.69 

.005 

11 

24* 

0 

0 

0 

107 

.89 

.315 

31* 

3 

1 

0 

0 

108 

.57 

.505 

1 

13 

20* 

1 

0 

109 

.83 

.658 

2 

29* 

4 

0 

0 

110 

.71 

.369 

2 

1 

5 

25* 

2 

111 

III 

.69 

1 

7 

3 

24* 

0 

112 

.69 

-.176 

1 

24* 

2 

1 

7 

113 

.89 

.401 

0 

0 

31* 

1 

3 

114 

.89 

.351 

2 

0 

1 

31* 

1 

115 

.94 

.284 

33* 

0 

2 

0 

0 

116* 

.97 

.348 

1       34*  0 
(F-0)  (G-0)  (H-0) 

0  0 
(I-O)  (J-0) 

117® 

.86 

.459 

2       0  1 
(F-1)  (G-0)  (H-1) 

29*  0 
(I-l)  (J-0) 

118* 

.91 

.484 

0  0 
(F- 32*) (G-0) 

0        0  1 
(H-1) (I-O)  (J-0) 

119* 

.89 

.479 

1  0 
(F-0)  (G-0) 

(H-0) 

1  1 
(I-30*)(J-1) 

320* 

.83 

.441 

1        0  1 
(F-0)  (G-0)  (H-3) 

0  29* 
(I-O)  (J-0) 

121* 

.94 

.353 

0 

0 

32* 

1 

0 

(F-0)  (G-0)  (H-0)  (I-O)  (J-1) 

♦correct  response 

^This  item  had  10  options,  A  -  J  woo 
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Table  16,  continued 


tern  Difficulty  Point  Number  Choosing  Each  Option 

No.  Level  BiseHal  r  A       B       C       D  E 


122 

.89 

.458 

1 

3 

31* 

0 

0 

123 

.91 

.232 

0 

32* 

0 

3 

0 

124 

.66 

.610 

5 

23* 

0 

3 

4 

125 

.80 

.209 

28* 

7 

0 

0 

0 

126 

.46 

.270 

16* 

19 

0 

0 

0 

127 

.37 

.206 

22 

13* 

0 

0 

0 

128 

.86 

-.015 

30* 

5 

0 

0 

0 

129 

.89 

.137 

4 

31* 

0 

0 

0 

130 

.86 

.543 

4 

30* 

0 

0 

0 

131 

.97 

.348 

1 

0 

34* 

0 

0 

132 

.91 

.321 

32* 

1 

1 

1 

0 

133 

.94 

.284 

0 

33* 

0 

1 

0 

134 

.20 

.290 

0 

1 

7 

26* 

0 

135 

.71 

.108 

2 

5 

25* 

1 

1 

136 

.91 

-.003 

0 

0 

2 

32* 

0 

137 

.57 

-.064 

20* 

6 

6 

2 

0 

Y=  105.54       S  =  12.59       Range  =  71  -  127        Reliability**  =  .878 


The  FORTAP  program  computes  a  Hoyt  reliability  coefficient. 
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Table  17 

Summary  of  Item  Analysis  of  Development  Test 

(N  =  36) 


Difficulty  Point  Nuinber  Choosing  Each  Option 

Level  Biserial  r  A       B       C       D  E 


1 

.94 

.389 

34* 

1 

0 

0 

0 

2 

.67 

.497 

10 

24* 

0 

0 

0 

3 

.75 

.169 

7 

27* 

0 

0 

0 

4 

.67 

.251 

24* 

11 

0 

0 

0 

5 

.92 

.470 

33* 

3 

0 

n 

6 

.64 

.331 

10 

23* 

0 

0 

n 

7 

.75 

.312 

5 

27* 

0 

n 

u 

8 

.89 

.248 

2 

32* 

0 

0 

0 

9 

.78 

.376 

28* 

4 

0 

0 

0 

10 

.9i7 

.295 

35* 

1 

0 

0 

0 

11 

.81 

.116 

1 

29* 

0 

0 

1** 

12 

.75 

.401 

6 

27* 

0 

0 

]** 

13 

.86 

.120 

31* 

4 

0 

0 

1** 

14 

.92 

.414 

2 

33* 

0 

0 

0 

15 

.83 

.428 

30* 

4 

0 

0 

0 

16 

.42 

.138 

18 

15* 

0 

0 

0 

17 

.94 

.086 

34* 

0 

0 

0 

18 

.94 

.187 

1 

34* 

0 

0 

0 

19 

.83 

.159 

2 

0 

30* 

4 

0 

20 

.92 

.442 

0 

0 

1 

1 

33* 

21 

.94 

.019 

2 

0 

0 

34* 

0 

22 

.  .89 

.101 

4 

32* 

0 

0 

0 

23 

.92 

.386 

33* 

3 

0 

0 

0 

24 

.94 

.154 

0 

1 

34* 

0 

0 

♦correct  response 

*nhe  same  person  marked,  option  E  for  questions  11,  12  and  13,  even  though  E  was 
not  a  legal  response. 
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Table  17,  continued 


T4*  Am 

Ui  TTi CUI  ty 

Or\\ 

roi  n  t 

Nunter  Choosing  Each  Option 

No. 

Level 

81  serial  r 

A 

8 

C 

0 

E 

CO 

Q1 

•  cSI 

OCA 

u 

o 

C 

c 

U 

CO 

OA 

-.Uoc 

1 

1 

U 

U 

Cl 

.4/ 

1  7>l 
.  1  /H 

1 

1  7* 

u 

lb 

U 

9fl 

U 

o 
c 

1  Q 
1  O 

U 

OO 

a 

•  cSo 

no  7 

0 

A 

u 

U 

A 

U 

flO 

9QQ 

32* 

0 

0 

3 

0 

31 

.97 

.201 

0 

0 

1 

35* 

0 

32 

.39 

.192 

11 

1 

14* 

8 

0 

33 

.97 

-.034 

1 

0 

0 

35* 

0 

34 

1 .00 

.000 

0 

0 

0 

36* 

0 

35 

.89 

.101 

2 

1 

•{ 

32* 

0 

36 

.78 

.357 

2 

4 

0 

28* 

0 

37 

.81 

.155 

2 

4 

29* 

0 

0 

38 

.58 

.331 

11 

21* 

1 

1 

0 

39 

.69 

.303 

25* 

0 

8 

1 

0 

40 

.94 

.389 

0 

34* 

1 

0 

0 

7=  32.28 

S  =  3.60 

Range  =25-38 

Reliability' 

a  _ 

.605 

*The  FORTAP  program  computes  a  Hoyt  reliability  coefficient. 
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Three  cautions  should  be  kept  In  mind  In  interpreting  the  data  from 
these  Item  analyses.    Each  Is  discussed  briefly  below. 

1.  The  only  Intent  in  conducting  this  analysis  was  to  provide 
preliminary  results  to  the  item  writers  to  enable  them  to  revise  items  as 
necessary.    Further  item  analysis  on  more  representative  samples  of  examinees 
will  be  necessary  before  stable  item  data  can  be  expected. 

2.  In  the  absence  of  definitive  demographic  data  about  members  of 
the  sample,  it  is  difficult  to  know  how  to  interpret  these  item  data.  For 
example,  items  which  appeared  too  simple  because  all  or  most  examinees  res- 
ponded to  them  correctly  may  merely  reflect  the  fact  that  the  76  examinees 
used  herein  are  unusually  competent.   Without  direct  data  on  competency  from 
other  sources,  such  issues  cannot  be  satisfactorily  resolved. 

3.  The  reliability  coefficients  recorded  above  apply  to  the  pools  of 
items,  not  to  the  final  subsets  of  items  chosen  from  these  pools  for  inclusion 
in  the  final  pilot  tests. ^ 

Results  of  the  item  analyses  were  sent  to  Millman  and  Page  (research 
and  evaluation  items)  and  Popham  (development  items),  along  with  all  comments 
of  examinees.   These  were  used  as  a  basis  for  further  revision  or  elimination 


of  items.    Some  items  which  failed  to  discriminate  were  retained  for  three 


It  should  be  noted  in  passing  that  traditional  concern  over  reliability, 
most  appropriate  for  tests  designed  to  diagnose  and  measure  individual  achieve- 
ment, is  relatively  less  important  here.   The  concern  is  not  with  the  accuracy 
with  which  we  measure  a  person's  true  score  but  rather  with  the  accuracy  with 
which  w<*  are  able  to  assess  the  profession's  ability  to  respond  correctly  to  a 
particular  item  (somewhat  analagous  to  the  problems  facing  National  Assessment). 
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Each  Item  In  the  Test  of  Educational  Research  and  Evaluation  was 
classified  according  to  Its  primary  and  secondary  emphasis  In  the  following 
three  categories: 

(1)  Research  skills  (designated  as  R) 

(2)  Evaluation  skills  (designated  as  E) 

(3)  Those  areas  of  knowledge  (of  statistics,  experimental  design 
and  psychometrics,  for  example)  which  are  essential  for  both 
research  and  evaluation  (designated  as  R  -  E). 

The  final  distribution  of  items  in  terms  of  primary  and  secondary  emphasis 

within  each  of  these  categories  is  as  follows:    Research  skills ,  23  Items 

with  primary  emphasis  and  6  items  with  secondary  emphasis;  Evaluation 

skills,   5  Items  with  primary  emphasis  and  1  Item  with  secondary  emphasis; 

and  Research-Evaluation  knowledge >  67  Items  with  primary  emphasis  and  35 

g 

Items  with  secondary  emphasis.     Summing  across  categories,  this  resulted 
in  90  items  with  primary  emphasis  in  research,  41  with  secondary  emphasis 
in  research,  72  with  primary  emphasis  In  evaluation,  and  36  with  secondary 
emphasis  in  evaluation. 

Within  these  broad  categories.  Items  were  also  classified  as  to 
specific  competencies  they  measured.   The  final  distribution  of  Items  on 
specific  competencies  is  shown  below  In  terms  of  primary  and  secondary 
emphases  (the  latter  appears  In  parentheses  throughout  this  list). 


The  number  of  itens  listed  above  exceeds  90,  the  nurriber  of  items 
in  the  tests,  since  some  Items  measured  more  than  one  competency  and  were 
classified  as  having  primary  emphasis  in  more  than  one  category. 
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Nuirber  of  items  with: 

primary  secondary 
emphasis  enphasis 


Research  Skills 

R  1*  10 

R  2  10 

R  3  3  0 

R  4  4  0 

R  5  2  0 

R6  3  1 

R  7  10 

R  8  3  1 

R  9  5  4 

Evaluation  Skills 

El  1  0 

E  2  1  0 

E  3  11 

E  4  10 

Knowledge  of  Research  -  Evaluation 
Statistics: 

SI  19  3 

'  ;  2  0 

S3  0  1 

S  4  2  0 

S  5  18 

S  6  1  0 
Experimental  Design: 

D  1  10 

D  2                           ^  0  5 

D  3  10  1 

D  4      .  2  0 

D  5  1  0 


Descriptions  of  all  skills  and  knowledge  are  given  in  the  early  pages 
of  this  chapter. 
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Number  of  items  with 


primary  secondary 
emphasis  emphasis 

Psychometrics: 

PI  10 

P  2  4  1 

P  3  4  4 

P  4  10 

P  5  1  1 

P  6  12 

P  7  0  1 

P  8  0  1 

Measurement: 

Ml  4  0 

M  2  10 

M  3  3  0 

M  4  1  1 

MS  0  3 

M  6  1  3 


General : 
G  1 

G  2" 
G  3 


2  0 
2  0 
2  0 
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Use  of  the  Pilot  Tests 

It  should  be  reiterated  that  the  pilot  tests  described  herein  were 
not  designed  for  use  in  making  important  decisions  about  individuals. 
Although  persons  could  use  individual  items  from  this  test  (or  even  the 
entire  test)  for  that  purpose,  it  should  be  kept  in  mind  that  these  tests 
are  still  in  preliminary,  exploratory  stages  and  would  need  extensive 
research,  refinement  and  nonrring  before  such  use  would  be  completely 
warrantedl^  The  tests  are  intended  to  provide  a  basis  for  attempting  to 
assess  the  proficiency  of  groups  of  persons— even  the  profession  as  a 
whole— on  selected  competencies  viewed  as  essential  to  the  conduct  of 
educational  research,  evaluation  and,  to  a  lesser  extent,  development. 

In  use  of  the  Test  of  Educational  Development  competencies,  the 
reader  should  keep  in  mind  two  points  discussed  earlier:   (1)  the  test 
focuses  primarily  on  one  type  of  devel opment— the  development  of  instruc- 
tional products,  and  (2)  within  that  area,  there  is  serious  disagreement 
on  the  part  of  some  examinees  as  to  whether  the  test  measures  important 
competencies . 

Obviously,  the  tests  could  be  used  independently  or  combined  for 
administration,  depending  on  the  purposes  underlying  their  use.  Also, 
items  could  be  selected  to  measure  specific  competencies  or  classes  of 
competencies  if  such  sub-divisions  were  viewed  as  useful  by  research 
trainers,  funding  agents,  or  AERA.   To  facilitate  such  fle>  bility  in  use, 
a  key  to  the  competencies  measured  by  each  item  (in  terms  of  both  primary 
and  secondary  emphasis)  is  provided  in  Appendix  K. 


In  the  interim  before  adequate  norming  is  conducted,  persons 
who  administer  these  items  to  any  sizable  group  are  invited  to  send  item 
analyses  data,  along  with  a  detailed  discription  of  the  examinees,  to 
Blaine  R.  Worthen,  Laboratory  of  Educational  Research,  University  of 
Colorado,  Boulder,  Colorado   ^302.     j^g  ^ 
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A  SURVEY  OF  EXISTING  TRAINING  OPPORTUNITIES  IN 
EDUCATIONAL  RESEARCH  AND  RESEARCH-RELATED  AREAS 

In  response  to  the  need  for  summary  Information  on  training  oppor- 
tunltles  In  educational  research,  the  American  Educational  Research 
Association's  Task  Force  on  Research  Training  undertook  as  one  of  its 
objectives  the  identification  of  formal  training  programs  in  educational 
research  and  related  areas.   The  intention  of  the  Task  Force  was  described 
by  two  subobjectives: 

4.1  To  determine  how  many  formal  programs  exist  specifically  for 
training  RDDE  personnel  in  education. 

4.2  To  determine  the  characteristics  of  these  formal  training 
programs  ... 

The  effort  was  to  include  compilation  of  a  directory  of  educational 
research  and  research-related  training  programs  in  the  U.S.    This  chapter 
Is  an  account  of  the  Task  Force  effort  to  attain  this  objective  and  a  des- 
cription of  its  results. 

Procedures 

Development  of  Definitions 

Early  in  the  work  on  this  objective,  it  became  clear  that  a  careful 
definition  of  terms  was  of  utmost  importance.   Not  only  was  it  necessary 
to  define  "educational  research  and  research-related  areas"  in  a  way  which 
would  convey  adequate  meaning;  it  was  also  crucial  to  decide  in  advance 
what  constituted  a  "formal  training  program." 

For  the  purposes  of  tliis  study,  educational  research  and  research- 
related  areas  were  defined  as  the  following:    educational  research;  the 
design  and  development  of  instructional  products  and  programs;  the 
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dissemination  of  products  and  curricula  to  the  educational  community;  and 

the  evaluation  of  materials  and  programs,  both  as  part  of  the  development 

effort  and  as  an  ongoing  monitoring  process.   These  areas  are  jointly 

designated  as  educational  RDOE. 

The  attempt  to  define  formal  training  programs  raised  several 

questions.    Some  categories  were  easily  decided.    For  example,  a  Ph.D. 

degree  program  in  educational  research  methodology  and  design  clearly  fit; 

so  did  an  M.A.  curriculum  in  educational  product  development.    But  what 

about  an  education  program  including  a  sprinkling  of  research  methods  and 

statistics  courses  as  part  of  the  general  requirements  for  the  Ph.D.  degree? 

Would  the  recipient  of  a  Ph.D.  In  science  education  from  such  a  program  be 

described  as  having  received  formal  educational  research  training? 

The  questions  were  even  more  difficult  and  more  numerous  when  the 
training  activities  of  organizations  and  agencies  which  do  not  offer 
academic  degrees  were  considered.    A  semi-annual  six-week  institute  on 
evaluation  methods  and  techniques >  conducted  by  a  state  education  depart- 
ment for  its  Title  I  evaluation  staff,  was  obviously  an  activity  which 
shouldJ}e  Included.    Monthly  seminars  on  advanced  statistical  methods 
offered  by  an  educational  testing  organization  could  also  be  included 
comfortably.   On  the  other  hand,  how  should  one  deal  with  a  three-day 
workshop  on  evaluation  theory  given  once  by  a  school  study  council  for  its 
members?   And  what  of  the  whole  area  of  on-the-job   training   that  occurs 
when  an  employee's  exposed  to  new  problems  and  new  ideas  from  the  more 
experienced  professionals  around  him? 

These  and  many  other  questions  were  raised  by  the  Task  Force  staff 
in  establishing  a  definition  of  "formal  training  programs."  Consideration 
of  the  intended  product  of  this  objective,  along  with  a  review  of  the 
financial  and  manpower  resources  available  for  its  accomplishment,  led  to 
the  following  decisions:  '  ■iKf 
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1.  The  Institutions,  organizations  and  agencies  In  the  survey 
were  divided  Into  two  broad  categories:  those  which  grant 
academic  degrees  and  those  which  do  not. 

2.  For  the  degree  granting  Institutions,  a  formal  training  program 
In  educational  research  anrJ  research-related  areas  was  described 
as  a  program  with  an  Identifiable  field  of  specialization  In 
educational  research,  development,  diffusion  or  evaluation  which 
Is  equivalent  to  other  areas  of  specialization  In  the  school  of 
education.    We  did  not  wish  to  restrict  our  definition  to  tnose 
programs  with  an  Institutionally-designated  "major"  In  educa- 
tional research,  since  It  was  our  feeling  that  such  a  definition 
would  exclude  a  large  number  of  programs  which  do  provide 
substantial  research  training.   On  the  other  hand,  we  were  not 
persuaded  by  the  argument  that  a  Ph.D.  program  In  education  Is, 
by  definition,  a  research  program  --  especially  In  the  light  of 
convincing  evidence  that  only  a  small  proportion  of  Ph.D. 
recipients  go  on  to  do  research.^ 

3.  For  those  organizations  and  agencies  which  do  not  grant  academic 
degrees,  formal  training  opportunities  In  educational  RDDE  were 
described  as  those  training  activities  which  are  a  scheduled  and 
ongoing  part  of  the  overall  activities  of  the  organization.  No 
limitations  were  set  on  the  type  of  vehicle  employed.  Activities 
included  were  lectures,  workshops,  seminars.  Institutes,  confer- 
ences »  and  the  like.   Similarly,  no  requirements  were  given  for 


See  Heiss,  in  Buswell,  et.  al.  (1966). 
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the  frequency  or  duration  of  the  activities:    they  might  occur 
once  a  week  or  once  a  year;  they  might  last  for  one  or  several 
days,  or  for  several  weeks.    Our  definition  did  not  include, 
however,  the  less  structured,  day-to-day  training  experiences 
of  an  employee  which  take  place  naturally  through  his  exposure 
to  the  projects  and  people  around  him.   The  Task  Force  staff 
felt  that  such  experiences,  though  undeniably  valuable,  cannot 
be  adequately  and  meaningfully  described  for  the  purposes  of  v 
this  study. 

Identification  of  Organizations  to  be  Surveyed 

The  next  step  toward  attainment  of  this  objective  was  to  identify 
those  institutions,  organizations  and  agencies  which  offer  RDDE  training. 
Several  categories  of  agencies  to  be  included  in  the  survey  were  identified 
by  the  Task  Force  staff:   colleges  and  universities,  state  departments  of 
education,  USOE-supported  R  and  D  centers  and  regional  laboratories.  Federal 
agencies,  regional  education  and  school  study  councils,  professional 
education  associations  and  similar  groups,  and  private  research  and  develop- 
ment organizations. 

Within  some  categories  (e.g.,  state  education  departments,  labs  and 
R&D  centers)  all  the  agencies  in  the  category  could  easily  be  included  in 
the  survey.    For  the  college  and  university  category,  criteria  such  as  the 
type  of  degree  granted  and  number  of  students  enrolled  could  be  used  to 
exclude  institutions  very  unlikely  to  offer  RDDE  training  programs  (e.g., 
a  private  undergraduate  institution  with  an  enrollment  of  2,100  students), 
thus  reducing  the  institutions  to  be  surveyed  In  this  category  to  a  manage- 
able number.   The  great  majority  of  regional  education  and  school  study 
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councils  could  be  readily  identified  through  the  membership  roster  of  the 
National  School  Development  Council. 

For  the  remaining  three  categories,  however  --  Federal  agencies, 
professional  associations  and  private  R&D  organizations  —  it  seemed 
necessary  to  employ  some  method  of  logical  selection  in  order  to  identify 
the  groups  most  likely  to  conduct  research  training  activities.  Selections 
in  these  three  categories  were  made  by  the  Task  Force  staff  in  consultation 
with  associates  of  the  Laboratory  of  Educational  Research  at  the  University 
of  Colorado  and  representatives  of  USOE. 

The  resulting  categories  of  institutions  and  organizations  included 
in  the  Task  Force  survey  consisted  of  the  following: 

1.  Colleges  and  universities.    The  Education   Directory  for  1969-70 
was  searched  and  a  list  was  compiled  of  all  doctoral  degree  granting 
institutions  and  those  master's  degree  granting  institutions  with  student 
enrollments  above  2000.   The  approximately  475  institutions  on  this  list 
were  surveyed  for  graduate  training  programs  in  educational  RDDE. 

In  order  to  identify  the  small  number  of  institutions  with  under- 
graduate programs  in  educational  RDDE,  the  Task  Force  staff  contacted 
Dr.  Robert  Bargar  at  The  Ohio  State  University  and  Dr.  John  Feldhusen  at 
Purdue  University.   On  their  advice  and  knowledge  of  undergraduate  training 
it  was  decided  that  the  recent  study  by  Dr.  Ewaugh  Fields  of  Drexel  ' 
University  included  information  on  all  relevant  undergraduate  training  i 
programs  and  should  serve  as  the  basis  for  examing  undergraduate  programs. 

2.  State  departments  of  education.    All  50  state  departments  of 
education,  plus  the  education  departments  for  the  District  of  Columbia, 
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Puerto  Rico  and  the  Virgin  Islands  were  included  in  the  survey  of  states. 
(See  Appendix  i.) 

3.  USOE-supported  R&D  centers  and  regional  educational  laboratories. 
A  list  of  the  eleven  regional  laboratories  and  eight  R&D  centers  was 
furnished  by  the  Research  Training  Branch  in  USOE.    (This  list  is  shown  in 
Appendix  L.) 

4.  Regional  education  and  school  study  councils.    The  82-member 
roster  of  the  National  School  Development  Council,  which  was  revised  in 
May  1971,  was  used  to  identify  groups  in  this  category,  all  of  which  were 
included  in  the  survey.    (The  list  is  given  in  Appendix  L.) 

5.  Professional  education  associations.    A  list  of  associations 
likely  to  be  engaged  in  educational  research  training  was  compiled  by  the 
Task  Force  staff  and  associates  of  the  Laboratory  of  Educational  Research 
at  the  University  of  Colorado.   The  Education  Directory  listing  of 
professional  associations  was  reviewed  by  the  staff  for  additional  groups. 
Organizations  which  had  been  considered  likely,  but  which  were  known  by  one 
or  more  staff  members  not  to  be  engaged  in  RDDE  training  activities,  were 
deleted.   The  remaining  list  contained  the  names  of  eleven  professional 
associations.   (These  are  given  in  Appendix  L.) 

6.    Federal  agencies.   At  the  request  of  the  Task  Force,  personnel 
in  USOE  agreed  to  furnish  the  names  of  Federal  agencies  to  be  included  in 
the  survey.    Some  agencies  which  had  been  considered  likely  were  deleted 
following  telephone  conversations  with  OE  representatives.    A  list  of 
eight  agencies  and  offices,  primarily  within  USOE,  was  supplied  to  the 
Task  Force.    (The  list  appears  in  Appendix  L.) 
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7.   Private  research  and  development  organizations,   USOE  made 
available  to  the  Task  Force  a  list  of  industrial  research  laboratories 
whose  areas  of  research  activity  include  education  or  education- related 
fields.    The  Task  Force  staff  added  the  names  of  research  and  testing 
organizations  which  they  regarded  as  potential  sponsors  of  research  train- 
ing activities.    This  resulted  in  a  list  of  47  organizations  included  in 
the  survey.    (They  are  listed  in  Appendix  L.) 

Development  of  an  Instrument 

Having  determined  the  catesfories  of  institutions  to  be  surveyed, 
and  having  developed  a  list  of  institutions  and  organizations  for  each 
category,  the  Task  Force  staff  next  considered  methods  to  identify  the 
agencies  which  actually  provide  training  opportunities  in  educational  RDDE. 
It  was  clear  from  the  number  of  organizations  on  the  lists  —  some  700  in 
all  —  that  a  mail  questionnaire  would  have  to  be  employed.  Considerations 
in  developing  a  questionnaire  were  (a)  to  develop  an  instrument  which 
would  produce  a  good  rate  of  response,  and  (b)  to  construct  the  instrument  in 
such  a  way  that  It  would  elicit  comparable  information  from  all  respondents. 

The  development  of  instruments  for  the  survey  divided  logically  into 
two  parts.   The  institutions  and  organizations  to  be  contacted  were  already 
separated  into  those  which  grant  academic  degrees  and  those  which  do  not. 
It  was  expected  that  there  would  be  little  comparability  between  the  kinds 
of  training  offered  by  the  two  groups.    Generally,  training  programs  in 
the  first  group  would  be  longer  in  duration,  would  tend  to  have  formal 
entrance  requirements,  and  would  be  composed  of  a  combination  of  course 
work,  area  examinations,  practical  experience  and  perhaps  an  original 
research  project.   In  addition,  certain  standards  of  performance  in  each 
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component  would  be  required  for  completion  of  the  program.  Training 
activities  of  the  non-^degree  granting  organizations,  on  the  other  hand, 
would  usually  be  short-term  and  would  offer  in  addition  to  some  lectures 
and  seminars  various  workshops,  institutes  and  conferences.  These 
activities  would  be  less  likely  to  have  the  kinds  of  formal  prerequisites 
that  are  necessary  for  entrance  to  an  academic  degree  program.  Sucii 
considerations,  and  differences  in  the  kinds  of  information  desired  from 
institutions  in  the  two  categories,  led  to  the  development  of  two  separate 
questionnaires. 

The  questionnaire  for  degree  granting  institutions.    In  constructing 
the  questionnaire  for  colleges  and  universities  (Q^),  the  Task  Force  staff 
used  three  sources  of  information.    First,  the  study  of  educational  research 
programs  conducted  by  David  Krathwohl  (1965)  was  reviewed.    An  attempt  was 
made  to  obtain  the  questionnaire  and  the  raw  data  used  in  the  Krathwohl 
study;  unfortunately,  these  were  not  available  and  a  replication  was 
therefore  impossible.    Secondly,  the  staff  reviewed  data  collected  for  an 
earlier  Task  Force  study  of  the  89  training  programs  established  under 
Title  IV  of  the  Elementary  and  Secondary  Education  Act.  Third, 
USOE  was  asked  to  indicate  specific  types  of  information 
considered  appropriate  for  the  directory  of  educational  research  training 
programs • 

^         A  draft  questionnaire  was  constructed  based  on  input  from  the  outside 
sources  and  containing,  in  addition,  a  number  of  items  written  by  the  staff 
to  solicit  information  not  obtained  in  earlier  studies.    The  draft  was 
then  submitted  to  associates  of  the  Laboratory  of  Educational  Research  (LER) 
for  their  reaction.    Following  their  criticisms  and  advice,  the  Task  Force 
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staff  prepared  a  revised  questionnaire  to  be  submitted  to  USOE  for  approval. 
The  suggestions  of  USOE  personnel  were  Incorporated  Into  a  final  one-page 
(two  sides)  Instrument  used  In  this  portion  of  the  survey.    The  final 
questionnaire  and  cover  letter  are  shown  In  Appendix  M. 

The  questionnaire  for  non-degree  granting  organizations.  Development 
of  the  questionnaire  for  non*degree  granting  Institutions  and  organizations 
(Q  )  proceeded  somewhat  differently.    At  the  beginning,  the  staff  found 
that  Its  conception  of  the  kinds  of  training  activity  conducted  by  these 
groups  was  much  less  clear »  and  that  there  was  less  certainty  about  the 
kinds  of  Information  to  seek  from  them.    It  was  therefore  decided  to  send 
a  letter  request  to  a  number  of  the  non-degree  granting  Institutions  in 
the  list.    The  letter  asked  that  the  addressee  supply  Information  on  any 
RDDE  training  activities  of  his  organization  and  that  he  furnish  the  names 
of  Individuals  within  the  organization  who  could  be  contacted  for  more 
detailed  data.   These  letters,  a  sample  of  which  appears  In  Appendix  M, 
were  sent  to  some  25  professional  associations  and  private  research  organi- 
zations, to  the  director  of  each  USOE  R&D  center  and  regional  educational 
laboratory  and  to  the  research  director  of  each  state  department  of  educa- 
tion.   Replies  were  received  from  74  of  the  97  groups,  some  of  which 
responded   that  they  do  offer  educational  RDDE  training  activities  of  some 
kind.   The  descriptive  information  furnished  by  the  respondents  was  used 
by  the  Task  Force  staff  in  developing  items  for  a  draft  questionnaire. 

From  this  point  on,  the  construction  and  apf^oval  of  the  instrument 
paralleled  that  of  the  questionnaire  for  degree  granting  institutions.  The 
final  questionnaire  appears  with  its  cover  letter  in  Appendix  M. 
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Distribution  of  the  Questionnaires 

On  July  16,  1971,  the  first  questionnaire  (Q^  was  mailed  to  478 
degree  granting  institutions;  208  of  these  were  doctoral  institutions 
and  270  were  institutions  which  grant  the  master's  but  not  the  doctorate. 
The  questionnaires  were  addressed  to  the  highest  administrative  officer 
(dean  or  chairman)  of  the  college,  department  or  school  of  education  or 
(in  the  absence  of  an  education  unit)  of  the  graduate  school.    By  search- 
ing the  Education  Directory,  the  institutional  catalogues,  and  miscellaneous 
sources,  the  Task  Force  staff  was  able  to  identify  by  name  about  60  percent 
of  the  addressees.    In  these  instances,  the  name  of  the  addressee  was 
typed  on  the  envelope,  along  with  his  title  and  business  address;  the  name 
was  also  handwritten  at  the  top  of  the  questionnaire  cover  letter.  In 
cases  where  the  individual's  name  was  not  known,  the  questionnaire  envelope 
was  addressed  by  title  only.  - 

In  all  instances,  the  mailing  consisted  of  the  questionnaire  itself, 
the  attached  cover  letter,  and  a  return  envelope,  pre-stamped  and  pre- 
addressed  to  the  Task  Force  office.    Both  the  initial  mailing  and  the 
return  were  by  first-class  mail. 

ft 

The  second  questionnaire  (Q  )  —  for  non-degree  granting  institutions 
and  organizations  —  was  mailed  on  August  5,  1971,  to  a  total  of  193  groups 
in  the  following  categories: 

1.   State  departments  of  education.   Q  was  not  sent  to  those 
departments  where  the  research  director  had  answered  our  earlier  inquiry 
by  replying  that  his  organization  offered  no  training  in  research  or 
research-related  areas.    (These  are  the  starred  entries  on  the  list  of 
state  education  departments  in  Appendix  L.)       was  sent  to  the  research 
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director  of  each  department  which  did  not  reply  to  our  earlier  inquiry. 

For  departments  which  did  reply  to  tne  initial  inquiry  by  providing  names 

2 

of  persons  to  contact  for  more  detailed  informetion,  Q  was  sent  to  those 

2 

individuals.    In  all,  Q  was  sent  to  54  individuals  in  37  state  education 
departments.  ; 

2.  USOE-supported  R&D  centers  and  regional  educational  laboratories, 
Q   was  sent  to  the  director  of  each  center  and  laboratory  wnether  or  not 

he  had  replied  to  the  earlier  letter  request.   There  were  19  individuals 
in  this  mailing. 

2 

3.  Regional  education  and  school  study  councils.    Q  was  sent  to 

all  82  members  of  the  National  School  Development  Council  (see  Appendix  L). 

In  all  cases,  the  questionnaire  was  addressed  to  the  director,  executive 

2 

secretary  or  president,  as  listed  on  the  membership  roster. 

2 

4.  Professional  education  associations.   Q  was  sent  to  the 
executive  officers  of  three  professional  associations  on  the  basis  of 
replies  to  the  Initial  letter  of  inquiry.    (The  associations  not  receiving 


because  they  Indicated  that  they  had  no  RDDE  training  are  starred  on 

3 

the  list  In  Appendix  L.) 

2 

5.    Private  research  and  development  organizations.    Q  was  sent 
to  all  organizations  In  this  category  which  are  listed  in  Appendix  L  except 
those  three  (starred)  which  replied  negatively  to  the  initial  letter 
request.    The  questionnaires  were  addressed  to  the  personnel  training 
directors  of  the  remaining  44  organizations. 

2 

The  NSDC  membership  roster  was  kindly  furnished  by  John  W.  Kohl, 
Executive  Director  of  the  organization. 

3 

In  addition,  the  Task  Force  staff  already  had  information  on  tne 
training  activities  of  several  otner  professional  organizations,  obtained 
for  Technical  Paper  No.  11  in  the  Task  Force  series.   This  information  was 
to  be  Included  in  the  training  directory  produced  by  the  Task  Force. 
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6.    Federal  agencies,    Q  was  sent  to  each  of  the  eight  Federal 
agency  personnel  suggested  oy  USOE  personnel    (see  Appendix  L). 

2 

In  all,  then,  Q  was  sent  to  210  individuals  representing  193 
separate  non-degree  granting  institutions  or  agencies.    As  with  q\  the 
mailing  consisted  of  the  questionnaire,  the  cover  letter,  and  a  pre- 
stamped,  pre-addressed  return  envelope. 

Follow-up  Procedures 

By  August  9,  the  deadline  indicated  in  the  cover  letter,  replies 
to  Ql^had  been  received  from  276  recipients  of  the  questionnaire  (58  per- 
cent).   A  follow-up  letter  was  prepared  and  was  mailed  on  August  13  to 
those  institutions  which  had  not  replied  at  that  time.   The  mailing 
included  the  follow-up  lettar  (see  Appendix  M),  another  copy  of  the 
questionnaire,  and  another  stamped,  addressed  return  envelope.    By  the 
time  of  the  cut-off  date  for  replies  (September  10),  129  additional 
institutions  had  responded,  bringing  the  total  number  of  respondents  to 
405  degree  granting  institutions  (85  percent  of  the  original  recipients 
of  the  questionnaire). 

For  the  193  non-degree  granting  organizations,  the  indicated  dead- 
line  was  August  16,  by  which  time  85  groups  (44  percent)  had  replied.  A 

follow-up  letter  was  prepared  (see  Appendix  M)  and  was  mailed  on  August  26 

2 

to  the  Q   recipients  who  had  not  responded.    This  mailing  also  included 
another  questionnaire  and  a  stamped,  addressed  return  envelope.  An 
additional  56  institutions  had  responded  by  the  September  17  cut-off  date, 
bringing  the  total  to  141  non-degree  granting  organizations  (73  percent). 
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Nonrespondents 

The  73  nonrespondents  to      have  the  following  cnaracteri sties: 

1.  Thirty-eight  nonrespondents  are  master's  degree  granting 
institutions;  35  are  doctoral  institutions.    The  master's  institutions 
represent  23  states;  the  doctoral  institutions  represent  20  states  and 
districts.    Among  the  master's  nonrespondents,  14  of  the  38  questionnaires 
(37  percent)  were  addressed  to  the  dean  of  the  school  of  education  by 
name,  rather  than  by  title  alone;  for  the  doctoral  nonrespondents,  tne 
ratio  is  21  out  of  35  (60  percent).    It  is  interesting  to  note  tnat  13 

of  the  doctoral  nonrespondents  are  institutions  which,  at  one  time,  had 
educational  research  training  programs  funded  under  Title  IV  of  the 
Elanentary  and  Secondary  Education  Act. 

2.  Size  of  nonresponding  institutions: 


Master's  Doctoral 

Under  3,000                  13  (34%)  3  (  9%) 

3,000  -  5,000                12  (32%  3  (  9%) 

5,000  -  10,000               10  (26%)  11  (31%) 

Over  10,000                    3  (  8%)  18  (51%) 

38  (100.)  35  (100%) 
3.    Affiliation  of  nonresponding  institutions: 

Master's  Doctoral 

State  or  local              25  (66%)  16  (46%) 

Independent                   7  (inv:)  12  (34%) 

Church-sponsored           _6  (Ifi^.),  7  (20%) 

38  (1 00/..)  35  (100%) 


The  52  non-degree*^granting  institutions  and  organizations  whicn 
did  not  respond  to  Q   are  distributod  follows: 

6   State  departments  of  education  (16  percent  of  tiie  37  groups 
surveyed  in  this  category) 

3    R&D  centers  and  laboratoriL' .  (lb  |i.rciMit  nf  tiu.<  19  groups 
surveyed) 
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27    Regional  education  councils  (33  percent  of  the  82  groups 
surveyed) 

14   Industrial  research  organizations  (32  percent  of  the  44  groups 
surveyed) 

2   Professional  associations  (67  percent  of  the  3  groups  surveyed) 
It  is  evident  from  the  above  findings  that  the  most  serious  nonresponse 
problem  was  in  industrial  organizations  and  regional  councils. 

Findings 

Degree  Granting  Institutions 

Of  the  405  degree  granting  institutions  responding  to  q\  91  indicated 
that  they  have  formal  training  programs  in  educational  research  or  research- 
related  areas.    Examination  of  tne  questionnaires,  however,  led  the  Task 
Force  staff  to  exclude  eight  of  these  from  consideration  in  t.ie  descriptive 
information  which  follows.    It  was  clear  from  the  information  given  uy 
these  eight  respondents  that  they  regard  a]!  of  their  students  as  educational 
research  trainees,  no  matter  what  their  field  of  specialization.  Since 
this  assumption  runs  counter  to  the  stated  definition  of  RDDE  trai.nng,  t.ie 
staff  deleted  the  eight  responses. 

The  remaining  83  positive  responses  from  institutions  in  3G  states 
are  included  in  the  following  data;  12  are  from  institutions  which  offer 
the  master's  degree  but  not  the  doctorate;  71  are  from  institutions  which 
offer  the  doctorate  (though  not  necessarily  in  educational  research). 
Thirty  of  the  83  positive  responses  are  from  institutions  which  have  oeen 
included  in  the  ESEA  Title  IV  graduate  r-esoarch  training  program. 
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Of  the  83  institutions  represented,  only  18  are  not  supported  by 

public  funds;  11  of  those  18  are  in  Massachusetts,  New  York  or  Pennsylvania. 

The  total  student  enrollment  of  the  83  institutions,  as  given  in  the 

1969-70  Education  Directory,  is  as  follows: 

Total  Enrollment         Number  of  Institutions 
<  3,000  2 
3,000  -  5,000  7 
5,000  -  10,000  15 
10,000  -  20,000  33 
>  20,000  26 

83 

The  following  descriptive  information  is  taken  directly  from  the 
questionnaire  responses  of  the  83  institutions.    The  number  to  the  left 
of  each  heading  corresponds  to  the  item  on  the  questionnaire.    (Refer  to 
Appendix  M  for  the  precise  wording  of  each  item.) 

2.   Number  of  years  the  program  has  been  in  existence 


1 
2 
3 
4 
5 
6 


5  7:3  (N  =  83) 

10  8:4 

8  9:1 

8  10  :  1 

25  >  10  :  6 

7  no  response  :  5 


3.   Degrees  offered  in  the  educational  RODE  program 

M.A.:  29  Ed.D. :  28  (N  =  83)* 

M.S.:  20  Ph.D. :  48^ 

M.Ed.:  25  Other:  }r 


4 

The  number  of  degrees  listed  exceeds  83  since  many  institutions 
offer  more  than  one  type  of  degree  in  educational  RDDE. 

^"Other"  includes  Education  Specialist  (4),  Advanced  Certificate  (2), 
Professional  Diploma  (2),  MAT  (1),  DA  (1),  Post-Master's  Diploma  (1),  and 
Research  Minor  (1). 
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For  those  institutions  wiiich  offer  the  doctorate: 

1  offers  the  D.A.  only 

6  offer  the  Ed.D.  only 

26  offer  the  Ph.D.  only 

22  offer  both  tne  Ed.D.  and  Ph.D. 

55 

Overall,  the  83  research  training  programs  may  be  categorized 
as  follows: 

28  less  than  the  doctorate 

18  doctorate  only 

35  doctorate  plus  master's 

_2  doctorate  plus  otner 

83 
it 

Number  of  students  in  educational  RDK 

Less  than  doctoral  level: 

,     '                     Full  time  340 

Part  time  367 

Doctoral  level : 

Full  time  656 

Part  time  397 

Criteria  for  admission  to  the  educational  RDDE  program 

Graduate  Record  Examination  (GRE) 

GRE  scores  are  considered  of  primary  importance  by  37 

p^-ograms.    For  the  29  which  indicated  desired  minimum  scores, 

the  ranges  and  medians  are: 

GRE  Total:  range  900-1300,  median  1000 

GRE  Verbal :  range  450-600,  median  500 

GRE  Quantitative:     range  450-650,  median  550 

An  additional  26  programs  consider  GRE  scores  of  secondary 
importance  for  admission;  tiie  minimum  scores  indicated  for 
these  programs  have  a  soinowhat  lower  median. 

Two  points  shouiti  be  made  regarding  the  information  on 
GRE  scores  as  criteria  for  admission.    First,  .uverul  insti- 
tutions wh'icn  considi.'r  the  verbal  >core  uiid  lite  total  score 
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of  secondary  Importance  regard  the  quantitative  score  as  a 
primary  admissions  Indicator.   Second,  many  programs  stress 
that  the  minimum  scores  which  they  listed  should  be  considered 
as  Indicators  of  the  desired  level,  rather  than  as  absolute 
cut-off  points  for  admission. 

Miller  Analogies  Test  (MAT) 

The  MAT  Is  of  primary  Importance  for  19  programs  and  of 
secondary  Importance  for  21.   For  those  listing  minimum  desirable 
scores,  the  range  of  raw  scores  Is  50-70  and  the  median  is  55. 

Other  Standardized  Tests 

Only  13  programs  listed  other  tests  as  criteria  for 
admission.   The  tests  listed  are  the  National  Teachers 
Examination  (3),  the  Advanced  GRE  in  Education  (3),  the  Terman 
Concept  Mastery  Test  (2),  the  Uatson-Glaser  (2),  Coop  English 
Test  (2),  the  STEP  Writing  Test  (1),  the  Doppelt  Test  of  Mathematical 
Reasoning  (1),  and  SCAT  (1). 

Grade  Point  Averages  (GPA) 

The  undergraduate  grade  point  average  is  of  primary 
importance  for  admission  to  54  programs.    For  the  48  which 
listed  a  generally  required  minimum,  the  median  GPA  was  3.0  on 
a  4-point  scale.   An  additional  20  programs  consider  the  under- 
graduate GPA  of  secondary  Importance.    It  should  be  noted  that 
the  minimum  GPA  required  was  often  qualified  as  applying  to  the 
last  two  years  of  undergraduate  work.. 
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Forty-six  programs  consider  the  graduate  GPA  of  primary 
Importance  and  38  of  these  Indicated  a  required  minimum  (median 
graduate  GPA  3.3  on  a  4-po1nt  scale).    Another  17  regard  the 
graduate  GPA  as  secondary  In  Importance. 

It  should  be  noted  here,  as  In  the  paragraphs  above  on 
GRE  scores,  that  the  listed  minimum  grade  point  averages  are 
regarded  as  Indicators  of  admissibility  and  not  as  absolute 
standards  for  admission. 

Previous  Course  Work 

Previous  course  work  In  particular  areas  Is  of  primary 
importance  for  22  programs  and  of  secondary  Importance  for  24. 
The  courses  listed  are  categorized  as  follows:    32  In  mathe- 
matics, statistics,  or  other  quantitative  work;  24  in  psycnology 
12  in  education.    (The  total  is  greater  than  the  number  of 
programs  responding  since  some  respondents  listed  more  than  one 
area  of  desired  course  work.) 

Undergraduate  Major 

Undergraduate  major  is  of  primary  or  secondary  Importance 
for  14  programs.    The  desired  majors  are:    10  psychology, 
9  mathematics,  5  education.    (The  total  is  greater  than  14  since 
some  respondents  listed  more  than  one  major.) 

Recommendations,  Experience,  Interview,  Interest,  Age 

Responses  to  the  remaining  six  categories  of  admission 
criteria  are  as  follows: 
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Primary 
Importance 


Secondary 
Importance 


No  Importance 
or  Blank 


Recomnendatlons 
Research  Experience 
School  Experience 
Interview 

Stated  Interest  In  Research 
Age 


40 
8 
13 
35 
65 
5 


34 
45 
37 

23 
12 
28 


9 

30 
33 
25 
6 
50 


Minimum  registration  requirements 

The  responses  to  item  6  of  the  questionnaire  were  collated 
to  produce  the  information  which  follows.   Nearly  all  respondents 
indicated  that  part-time  students  are  accepted  in  the  research 
training  program;  only  four  stated  that  they  have  no  part-time 
students.    For  the  79  institutions  accepting  part-time  students, 
the  minimum  number  of  credit  hours  required  per  academic  term  is 
given  below. 

Credit  Hours  Required 


It  should  be  pointed  out  that,  in  most  cases,  nine  or  more  credit 
hours  per  term  is  equivalent  to  full-time  registration.  There- 
fore it  is  probably  more  accurate  to  say  that  68  programs  — 
rather  than  79  —  accept  part-time  students,  and  that  15  do  not. 

In  terms  of  the  minimum  number  of  courses  required  per 
term,  37  programs  require  one  course,  10  require  two  courses, 
and  7  require  three  or  four.    (Since  several  respondents  indicated 
a  minimum  number  of  hours  but  not  a  minimum  number  of  courses. 


Per  Academic  Term 


Number  of  Programs 


1  to  3 
4  or  5 
6  to  8 


9  or  more 
no  minimum 


34 
10 
11 
11 
13 
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the  total  here  does  not  agree  with  that  above.)   Again, "three 
or  four  courses  per  term  is  essentially  a  full-time  requirement. 

Of  the  83  institutions  responding,  24  operate  on  the  quarter 
system,  54  on  the  semester  system,  and  1  on  trimesters;  4  insti- 
tutions left  the  item  blank. 
Minimum  period  of  full-time  residence  required 

All  except  3  of  the  55  programs  which  offer  the  doctorate 
require  full-time  registration  of  their  students  for  some  period 
of  time. 

Minimum  Period  of 
Full-time  Registration       Number  of  Doctoral  Programs 

less  than  1  year  2 

1  year  42 

2  years  8 

For  the  28  sub-doctoral  programs,  on  the  other  hand,  only 
5  require  some  period  of  full-time  registration.    Four  of  these 
specify  one  full-time  semester;  the  fifth  requires  three  full- 
time  quarters. 

Areas  of  emphasis  in  RDDE  training  programs 

The  areas  of  emphasis  of  the  83  RDDE  training  programs  are 
listed  below.    Since  most  programs,  indicated  more  than  one  area, 
the  total  is  much  greater  than  83. 
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Area  of  Emphasis  Number  of  Programs 

Research  methodology/deslgn/statlstlcs  71 

Measurement  51 

Evaluation  47 

Psychology  22 

Curriculum  and  instruction  16 

Administration  12 

Counseling  and  guidanceg  6 

Education  subject  areas^  6 

History  and  philosophy  of  education  3 

Vocational  education  1 

Special  education  1 

Educational  product  or  program  development  10 

Dissemination  and  diffusion  4 

0ther7  9 


It  should  be  noted  that  in  item  8  on  the  questionnaire  the 
respondent  was  asked  to  indicate  those  areas  in  which 
"researchers  and  research-related  personnel"  are  being  prepared. 
The  above  listing  may  thus  be  taken  to  indicate,  for  example, 
that  12  institutions  offer  training  in  research  in  administra- 
tion, and  that  6  programs  emphasize  research  in  counseling  and 
guidance. 

Although  it  is  not  evident  from  the  figures  above,  it  is 
interesting  to  note  that  nearly  half  the  programs  (39  out  of  83) 
indicate  research  methodology,  measurement,  or  evalution  as  their 
only  major  areas  of  empnasis.    An  additional  ten  programs  focus 
on  one  or  more  of  those  three  areas  plus  psychology.  Thus, 
only  34  of  the  83  programs  have  one  or  more  RDDE  specializations 


Subject  areas  include  reading,  scicMice,  matliematics,  social 
studies ,  English,  foreign  languages. 

^•'Other"  includes  com(>uter  science,  urban  schools,  policy  studies. 
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outside  the  areas  of  research  methodology,,  iiK^asurEment, 
evaluation,  and  psychology. 
Course  requirements 

There  is  great  variation  in  both  the  number  and  kind  of 
course  requirements  among  the  responding  institutions.  The 
responses  to  this  item  for  master's  and  doctoral  programs  are 
given  below.    (It  should  be  noted  that  several  institutions 
responded  only  by  checking  the  areas  in  which  courses  are 
required,  rather  thah  by  indicating  the  number  of  required 
courses  in  each  area.    In  these  cases,  the  institution  was 
recorded  as  requiring  one  course  in  the  indicated  area.) 
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TABLE  19 

Course  Requirements  In  Educational  RDDE  Programs 


Area 


Introduction  to  educational 
research 

Advanced  research  methods 

Statistics 

Computer  methods  and  use 
Testing  (use  of  standardized  tests) 
Measurement  and  test  construction 
Evaluation 

Curriculum  and  Instruction 
Administration 
Research  management 
Communications 
Marketing 

Politics/finance  of  education 

Educational  change 

Product  development 

Instructional  sequencing 

Psychology  (Including  educational 
psychology) 

Sociology 

History/philosophy  of  education 
Other^  r 


Number  of  Courses  Required 
Master's  Programs       Doctoral  Programs 

1      2      3+  1      2  3+ 


46 

3 

0 

33 

3 

1 

27 

3 

0 

24 

20 

5 

27 

20 

6 

13 

8 

32 

20 

3 

1 

25 

8 

3 

32 

3 

2 

22 

0 

30 

4 

0 

25 

11 

3 

15 

0 

0 

23 

0 

14 

2 

2 

16 

2 

3 

0 

1 

5 

2 

2 

0 

0 

9 

0 

1 

0 

0 

2 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

3 

2 

0 

1 

0 

0 

11 

0 

1 

1 

0 

1 

2 

0 

1 

5 

2 

0 

5 

4 

0 

18 

6 

7 

11 

4 

13 

6 

0 

0 

8 

0 

0 

25 

1 

1 

18 

3 

1 

3 

0 

0 

2 

1 

1 

^"Other"  Includes  planning,  urban  studies,  philosophy  of  science,  learn- 
ing in  the  affective  domain,  research  in  related  disciplines. 
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10.    Practical  research  experience  required 

Of  the  83  institutions  responding  positively  to  the 
questionnaire,  46  require  at  least  one  of  the  kinds  of  practical 
experience  listed  in  item  10.    Thirteen  of  these  represent 
programs  which  do  not  offer  the  doctorate;  33  represent  programs 
which  do  offer  the  doctorate.    The  total  responses  are  as 
follows  (some  programs  require  more  than  one  kind  of  research 
experience): 

Type  of  Experience  Required^  Number  of  Programs 

-^Assistantship  or  internship  wi thin 
the  program: 

Across  several  projects  13 
On  a  particular  project  16 

Assistantship  or  internship  in  a 
campus  research  unit  outside  the 
program: 

Across  several  projects  2 
On  a  particular  project  7 

Assistantship  or  internship  off  campus^  8 

Apprenticeship  to  a  senior  researcher  10 

Technical  consultant  services  7 


Non-Degree  Granting  Organizations 

Of  the  141  non-degree  granting  institutions  and  organizations  which 
responded  to  Q^,  a  total  of  55  indicated  that  they  offer  RDDiI  training 
activities  which  fit  the  definitions  given  in  the  questionnaire.    The  55 
organizations  are  in  the  following  categories: 


An  additional  nine  programs  require  one  or  more  of  the  above  kinds 
of  experience,  but  the  type  was  not  specified.   These  nine  programs  are 
included  in  the  total  of  46. 

^Usually  in  a  school  setting  or  educational  research  office. 
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19  State  departments  of  education 

9   USOE-supported  R&D  centers  and  laboratories 
19   Regional  education  and  school  study  councils 

7  Private  research  and  development  organizations 
Federal  agency 


Because  of  the  great  variety  of  training  activities  In  educational 
research  and  research-related  areas  which  are  conducted  by  the  55  organi- 
zations  and  because  of  the  lack  of  uniformity  In  responses.  It  Is  not 
possible  to  give  precise  data  for  each  of  the  questionnaire  Items.  The 
following  Is  rather  an  attempt  to  present.  In  narrative  form,  a  summary 
description  of  training  activities  for  each  category  of  organizations. 
For  the  purposes  of  this  description,  the  activities  of  the  one  Federal 
office  —  a  USOE  bureau  ~  will  be  combined  with  those  of  the  R&D  centers 
and  laboratories. 

State  departments  of  education.    The  most  common  vehicles  for 
research  training  among  the  19  education  departments  are  workshops  and 
conferences  of  one  to  two  ds^s'  duration.   The  frequency  of  the  training 
activities  varies  from  monthly  to  annually,  with  nine  state  departments 
of  education  offering  only  annual  programs. 

About  one-third  of  the  departments  restrict  participation  to  their 
own  employees  and  another  third  permit  limited  outside  participation  (by 
teachers,  principals  and  school  superintendents,  for  example).    The  remain 
Ing  programs  are  open  to  anyone  Interested.   Only  one  education  department 
charges  a  fee  for  participation,  and  that  Is  minimal  ($2.50  for  reglstra- 
tlon).    Three  respondents  Indicated  that  support  for  their  programs  comes 
from  USOE  funds. 
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Nearly  all  of  the  education  departments  use  their  own  staff  as 
instructors  in  some  of  their  training  activities.   About  half  also  use 
persons  outside  the  organization. 

Regarding  major  objectives  of  the  organization  in  providing  training 
opportunities »  four  options  were  presented  in  the  questionnaire: 
(a)  presenting  information  on  new  developments »  (b)  raising  job  aspirations 
of  staff  members^  (c)  updating  knowledge  in  subjects  previously  studied, 
and  (d)  giving  in-depth  training  in  new  subjects.   The  major  objectives 
of  the  19  state  education  departments  are  divided  about  equally  among  these 
four. 

Nine  of  the  responding  education  departments  have  formal  internship 
arrangements  with  colleges  and  universities.    Ten  allow  staff  members  time 
off  with  pay  to  take  credit  courses  in  an  academic  institution;  only  four 
pay  the  employee's  tuition  for  such  courses. 

A  total  of  55  topics  for  training  seminars,  workshops,  institutes 
and  the  like  ~  held  over  the  past  year  or  planned  for  the  coming  year  — 
were  listed  by  the  19  organizations.    They  have  been  combined  into  five 
areas,  as  follows: 


^   *Statistical  analysis,  curriculum  innovation,  project 
development,  data  processing. 

R&D  Centers,  regional  laboratories,  USOE  bureau.    Most  of  the.  10 

organizations  in  this  category  hold  one-  to  three-day  seminars  or  workshops, 

some  monthly  or  quarterly  and  some  annually.    One  laboratory  conducts  an 


Topic  Area 


Number  of  Times  Listed 


Evaluation 

Management,  including  PPBS 
Proposal  de vel opment 
Dissemination 
Other* 


27 
11 
7 
6 
4 
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annual  seminar  of  one  to  two  weeks  duration  and  an  annual  workshop  one 
to  four  weeks  In  length.   Another  has  two  conferences  per  year,  each  three 
to  four  weeks  long.   One  of  the  centers  conducts  a  six-week  institute 
three  times  a  year. 

Two  organizations  restrict  participation  to  their  own  employees  and 
two  others  allow  participation  by  anyone  interested.    The  remaining  six 
accept  limited  outside  participants.    Two  of  the  R&D  centers  charge  fees 
for  their  training  activities  ($10  and  $25).    Another  indicates  that 
tuition  is  charged  for  those  training  activities  which  earn  graduate  credit 
at  a  participating  academic  institution. 

Seven  of  the  groups  use  only  their  own  staff  as  instructors  in  the 
training  program.   The  other  three  use  outside  instructors  as  well. 

Seven  respondents  indicated  that  the  major  objective  of  their 
programs  is  to  provide  in-depth  training  in  subjects  new  to  the  participants. 

One  stated  that  the  training  objective  is  to  allow  the  organization's 
constituents  to  participate  in  educational  development. 

Six  of  the  labs  and  R&D  centers  have  internship  programs  with 
colleges  and  universities.    Eight  allow  their  employees  time  off  with  pay 
for  course  work  and  four  of  these  also  pay  tuition. 

The  31  training  topics  listed  by  respondents  in  this  group  —  from 
activities  held  in  the  past  year  or  planned  for  the  coming  year  ~  are 
categorized  as  follows: 


Topic  Area  Number  of  Times  Listed 


Evaluation  13 

Product  or  program  development  6 

Dissemination  3 

Management  3 

Change  theory  and  strategies  3 

Other*  _3 


31 


*R&D  process,  proposal  writing,  problems  of  Appalachian  education. 
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Regional  education  and  school  study  councils.   The  most  comnon 
training  vehicles  of  the  19  regional  councils  are  again  workshops,  seminars 
and  conferences,  lasting  from  one  to  five  days  and  occurring  monthly, 
quarterly,  semi-annually  or  annually.    In  addition,  one  group  conducts  an 
annual  ten-day  institute  and  another  holds  two-week  workshops  continually 
throughout  the  year.    In  general,  training  activities  sponsored  by  the 
regional  councils  are  open  to  limited  outside  participants. 

Eight  groups  charge  no  fees  for  participation  in  their  training 
programs;  the  average  charged  by  the  remaining  eleven  groups  ranges  from 
$5  to  $60,  witn  a  median  of  $25. 

Nearly  all  of  the  groups  draw  their  instructional  staff  from  both 
inside  and  outside  the  organization.    About  half  the  respondents  from 
regional  councils  indicated  that  the  major  objective  of  their  training 
programs  is  to  inform  participants  of  new  developents  in  their  fields. 
The  other  organizations  regard  in-depth  training  in  new  subjects  as  their 
primary  objective.   Nine  of  the  councils  have  formal  internship  arrange- 
ments with  colleges  and  universities. 

Ten  organizations  indicated  that  their  employees  are  given  time  off 
with  pay  to  attend  courses  in  a  college  or  university;  of  tnese,  seven 
tuition  for  courses  taken  by  employees. 

The  43  research-related  training  topics  listed  by  regional  council 
respondents  have  been  combined  into  the  6  categories  below.    It  should  be 
pointed  out  that  many  respondents  also  listed  in  this  item  such  topics 
as  student  unrest,  drug  abuse,  human  relations,  and  health  and  safety. 
Since  these  are  not  considered  to  ho  oducal.ional  research-rolatod  topics, 
according  to  the  Task  Force  definitions,  they  are  not  included  in  this  listing 
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Topic  Area 


Number  of  Times  Listed 


Evaluation 

Management,  including  PPBS 
Proposal  development 
Dissemination 

Curriculum  and  instruction 
Other* 


20 
9 
3 
2 
2 
7 


43 


♦Research  design  and  techniques,  research  in  reading,  R&D  for 
Appalachian  schools,  statistical  methods. 

Private  research  and  development  organizations.    About  half  of  the 
seven  private  research  organizations  have  training  programs  which  include 
workshops  and  seminars  lasting  for  one  to  five  days.    Activities  of  greater 
duration  are  offered  by  three  groups:    one  conducts  one-  to  two-week  work- 
shops three  times  a  year;  the  others  have  annual  institutes  of  two  to  four 
weeks  and  two  to  three  months,  respectively. 

With  one  exception,  training  activities  of  these  groups  are  open  to 
those  outside  the  organization.   Five  organizations  charge  no  fees  unless 
college  credit  is  earned;  then  tuition  is  paid  to  the  academic  institution. 
Another  charges  an  average  fee  of  only  $2  per  person.    The  remaining 
organization  has  fees  ranging  from  $350  to  $650.   Most  respondents  indicate( 
that  funding  for  the  training  programs  comes  from  the  organization  itself. 

Four  of  the  organizations  use  only  their  own  staff  members  as 
training  instructors.  '  The  other  three  use  outside  persons  as  well.  All 
but  one  of  the  respondents  indicated  that  the  objective  of  the  training 
program  Is  to  Infonn  participants  of  new  developments  in  their  field  or  to 
update  their  training  in  subjects  they  have  a:l ready  studied. 

Four  of  the  private  R&D  organizations  have  e^her  formal  or  informal 

inlwnship  a^^  ^° 
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groups  neither  pay  tuition  nor  give  time  off  with  pay  for  college  or 

university  courses  taken  by  employees.    Three  groups  do  give  time  off  with 

/ 

psiy  and  two  of  these  also  pay  tuition. 

The  16  topics  for  training  activities  conducted  by  private  R&D 
organizations  in  the  past  year  or  planned  for  the  coming  year  have  been 
combined  into  the  6  categories  below. 

Topic  Area  Number  of  Times  Listed 

Research  design,  programming 

and  evaluation  2 

Product  and  program  development  4 

Dissemination  2  " 

Evaluation  _        '  3 

Measurement  4 
Statistics 

16  — 


Discussion 

The  information  gathered  in  the  course  of  this  stu4y  —  from  83 
degree  granting  institutions  and  55  other  organizations  and  agencies  — 
has  been  used  to  compile  a  directory  of  training  opportunities  in  educa- 
tional  research  and  research-related  areas.    (The  directory  appears  as  a 
separately  bound  appendix  to  this  report.)    It  is  the  hope  of  the  Task 
Force  that  this  directory  will  serve  as  a  valuable  guide  to  those  indi- 
viduals who  seek  such  training  and  as  a  source  of  information  to  those 
groups  concerned  wi th  expandi ng  and  improving  educational  RDDE  training 
in  the  United  States.   The  following  summary  and  discussion  of  some  of 

the  findings  may  be  of  benefit. 

vAnong  the  83  eduM^^  programs  in  colleges  and  universities, 

only  18  are  located  ^  non-public  institutions i  ne^      three-fourths  are  in 
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Institutions  with  student  enrollments  over  10,000.    With  the  exception  of 
1966-67,  between  five  and  ten  new  training  programs  have  been  started  each 
year  for  the  past  six  years.   The  much  larger  number  for  1966-67  reflects 
the  impetus  of  the  ESEA  Title  IV  research  training  provisions;  14  of  the 
25  programs  begun  in  that  year  are  in  institutions  which  received  Title  IV 
funds. 

Although  the  M.Ed,  and  Ed.D.  are  generally  not  considered- research 
degrees,  a  sizable  number  of  programs  offer  these  degrees  in  educational 
research  (25  and  28  programs,  respectively).   The  Ph.D., however,  is  by  far 
the  more  common  of  the  two  doctoral  degrees,  with  48  programs  offering  the 
Ph.D.  compared  to  28  for  the  Ed.D.    It  is  interesting  to  note  that  fully 
one-third  of  the  83  programs  do  not  offer  any  doctoral  degree,  but  concen- 
trate  Instead  on  master's  level  and  education  specialist  training  In 
educational  research. 

Nearly  all  of  the  doctoral  training  programs  (50  out  of  55)  require 
at  least  one  year  of  full-time  residence  for  completion  of  the  degree. 
The  great  majority  of  sub-doctoral  programs,  on  the  other  hand,  require 
no  full-time  registration  of  their  students. 

Sixty  percent  of  Jhe  doctoral  programs  require  formal  practical  . 
research  experience  of  their  graduates,  compared  to  slightly  under  half 
of  the  sub-doctoral  programs.    The  kind  of  experience  most  heavily  favored 
is  an  assis^antship  or  internship  on  one  or  more  projects  within  the 
gradUate>prognam  itself.    Only  ten  programs  require  formal  apprenticeship 
to  a  senior  researcher. 

At  the  end  of  the  1970-71  academic  year,  over  700  sub-doctoral 
students  were  enrolled  in  educational  RDDE  programs,  about  half  of  whom 
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were  studying  full  time;  over  60  percent  of  the  more  than  1000  doctoral 
students  were  engaged  in  full-time  study.    In  discussing  the  Title  IV 
graduate  research  training  programs,  Clark  and  Hopkins  (1969,  p.  305) 
indicated  that  about  one-half  of  the  full-time  sub-doctoral  students  and 
one-third  of  the  full-time  doctoral  students  could  be  expected  to  receive 
their  degree  each  year.    If  the  same  assumption  is  applied  to  the  registra- 
tion figures  obtained  in  this  study  —  and  if  it  is  assumed  further  that 
most  students  complete  their  program  and  enrollment  figures  will  not 
decline  substantially  —  then  the  existing  educational  research  programs 
are  producing  between  350  and  400  graduates  each  year  from  tiieir  full- 
time  students  alone. 

In  terms  of  areas  of  emphasis,  the  graduate  programs  in  this  study 
focus  overwhelmingly  on  research  methodology,  measurement  and  evaluation. 
Only  a  handful  of  programs  place  major  emphasis  on  educational  development 
or  dissemination  and  diffusion. 

The  same  is  true,  to  a  very  large  degree,  of  the  55  non-degree 
granting  organizations  which  regularly  offer  research-related  training. 
Evaluation  is  by  far  the  area  of  greatest  interest,  as  indicated  in  the 
workshop^,  seminar,  institute  and  conference  topics  listed  by  the  respondents. 
This  is  true  not  only  in  state  education  departments ,  where  ESEA  Title  I 
and  Title  III  evaluation  requirements  must  be  met,  but  also  in  R&D  centers 
and  laboratories  and  in  the  regional  education  councils.    The  areas  of 
management,  development  and  dissemination  fall  well  behind   in  the  topics 
listed.  ^ 

Somewhat  more  than  half  of  the  non-degree  granting  organizations 
consider  the  major  focus  of  their  training  program  to  be  the  ujjdating  of 
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earlier  training  rather  than  in-depth  instruction  in  areas  new  to  the 
participants.    The  lack  of  emphasis  on  in-depth  training  is  probably  also 
reflected  in  the  fact  that  only  eight  of  the  fifty-five  responding  organ- 
izations have  regular  training  activities  of  more  than  five  days  duration. 
It  is  also  true  that  over  half  of  the  organizations ,  through  internship 
arrangements  with  colleges  and  universities,  provide  opportunities  for 
practical  experience  in  educational  research  and  research-related  areas.  ' 
This  underscores  the  apparent  role  of  the  non-degree  granting  organizations 
as  facilitators  —  rather  than  major  instigators  —  of  research -related 
training.  • 

Since  the  major  purpose  of  the  effort  reported  in  this  chapter  was 
to  gather  information  for  the  directory  of  training  opportunities,  it  was 
npt  anticipated  that  the  data  which  have  been  described  would  yield  any 
major  conclusions.   Nevertheless,  three  very  general  comments  seem  appro- 
priate.   First,  the  colleges  and  universities  are  preparing  a  respectable 
number  of  students  in  research  methods  and  design,  measurement  and 
evaluation.   Second,  other  organizations  conduct  training  activities  in 
response  to  the  felt  need  of  their  staff  and  clients;  at  present,  the 
major  need  for  such  training  seems  to  be  in  the  area  of  educational  evalu- 
ation.   It  may  be  that  universities  and  colleges  need  still  more  emphasis 
on  training  in  this  area.    Last,  among  the  groups  surveyed  there  appears 
tQ»be  no  niaj or  effort  underway  for  the  training  of  educational  developers 
and  disseminators. 
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AN  UPDATING  OF  THE  CLARK-HOPKINS  MANPOWER  PROJECTIONS 


1 


1 
I 


Purpose  of  the  Stucly 

Educational  research  training  programs  have  as  their  major  purpose 
the  production  of  trained  personnel  to  carry  out  research  and  research- 
related  activities  In  education.    One  could  argue  that  training  program 
directors  have  an  obligation  to  both  the  existing  research  community  and 
Its  prospective  members  to  keep  informed  of  the  size  of  the  research 
community  so  that  neither  jobs  go  unfilled  nor  people  go  unemployed.  The 
most  effective  use  of  manpower  would  occur  If  precisely  enough  personnel 
were  trained  In  educational  research,  development,  diffusion,  and  evalua- 
tlon  (RDDE)  to  fill  existing  job  openings.    This  Is  not  at  all  practical, 
however,  because  the  educational  system  Is  constantly  changing,  resulting 
In  sometimes  drastic  reductions  or  Increases  In  the  need  for  trained  per- 
sonnel.    To  be  prepared  for  those  changes.  It  Is  necessary  to  make  projec- 
tions—Informed estimates  (based,  1t  Is  hoped,  on  valid  and  reliable  data) 
of  where  the  profession  1s  heading. 

Such  a  projection  was  made  In  1969  by  David  L.  Clark  and  John  E. 
Hopkins  In  A  Report  on  Educational  Research,  Development,  and  Diffusion  Man- 
power, 1964-1974  (Clark  and  Hopkins,  1969).    Since  that  time,  economic  and 
political  shifts  have  occurred  which  make  the  projection  overly  optimistic. 
Because  of  those  shifts,  the  AERA  Task  Force  felt  It  necessary  to  update  the 
earlier  projection  to  make  it=^confon|i  more  closely  to  reality.  Thus, 
one  of  the  objectives  of  this  year's  Task  Force  activities  was:  "To 
revise  existing  manpower  projections  based  on  recent  funding  of  RDDE  , 
activities"   (sub-objective  5.1) .     It  was  found  that  the  ^ 


iThic  chaoter  which  appeared  earlier  as  Technical  Paper  No.  25  in  the 
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Clark-Hopkins  projection  was  the  most  comprehensive— indeed  the  only 
financially-based—projection  of  the  demand  for  trained  personnel 
in  educational  RDDE.   Therefore,  this  chapter  is  an  updating  of  the 
projection  made  in  the  1969  report  based  on  actual  funding  through 
1971  and  current  administrative  feelings  on  the  future  of  RDDE  activi- 
ties through  1974. 

Very  briefly,  the  procedures  used  in  the  1969  Clark-Hopkins 
report  were  the  following.    Basic  data  regarding  personnel  supported  in 
RDD  positions  were  obtained  from  those  FY  1966  proposals  which  had  been 
approved  for  funding.    Financial  data  (appropriations,  expenditures) 
were  available  through  FY  1968,  in  most  cases.    Program  administrators 
in  eight  USOE  divisions  and  one  NSF  program  were  interviewed  to  obtain 
their  perceptions  of  the  growth  of  their  program  through  FY  1974.  Using 
this  data  base  and  logically  derived  extensions  of  the  data,  three  pro- 
jections were  made  of  anticipated  funding  in  FY  1974:  a  least  optimis- 
tic one,  a  most  optimistic  one,  and  a  rAost  likely  one.  Financial 
projections  were  transformed  into  personnel  projections  through  the  use 
of  a  "growth  ratio,"  the  amount  of  money  projected  for  1974  (taking  into 
account  inflationary  trends)  divided  by  the  amount  of  money  actually 
available  to  support  a  given  number  of  people  In  1968.    If  the  growth 
ratio  was  one,  exactly  the  same  nunter  of  people  could  be  supported  in 
1974  as  were  supported  in  1968.    Personnel  demand  was  projected  along 
three  dimensions:    institutional  setting,  professional  assignment,  and 
RDD  function. 

The*  manpower  projections  in  the  1969  Clark-Hopkins  report  have 
several  limitations.   These  incl^jide  the  following:  ' 
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1.  When  the  original  data  were  gathered,  evaluation  personnel 
were  not  identified  as  a  separate  group  as  were  research,  development, 
and  diffusion  personnel.    There  is  no  projection,  therefore,  of  the 
demand  for  or  influence  of  this  visible  and  important  group. 

2.  The  heacly  atmosphere  which  prevailed  after  passage  of  the 
Elementary  and  Secondary  Education  Act  (ESEA)  in  1965  resulted  in  some 
of  the  original  projections  being  so  optimistic  that  they  bear  little 
relationship  to  the  current  situation. 

3.  The  projections  were  based  on  the  number  and  type  of 
positions  listed  in  the  budgets  of  funded  proposals,  on  the  assumption  that 
the  project  directors  would  actually  employ  the  number  and  type  of 
personnel  cited  in  their  budgets.    No  follow-up  was  made,  however,  to 
determine  what  personnel  were  actually  employed  to  carry  out  the  projects 
and  thus  to  determine  the  validity  of  the  assumption. 

4.  The  possibility  of  retraining  professionals,  from  fields 
other  than  education  to  meet  the  supply  deficit  in  education  was  not 
fully  examined. 

A  full  replication  of  the  original  study  would  have  been  necessary 
to  remecly  the  first  of  the' above  limitations.   A  new  stuc|y  would  be 
needed  to  examine  fully  the  fourth  item.     Neither  the  time  nor  the 
money  available  to  complete  objective  5.1  would  permit  this.   The  present 
chapter  does  attempt  to  remedy  the  second  and  third  items  above  by: 


An  argument  against  retraining  as  a  major  source  of  supply  is 
presented  as  Appendix  II  to  this  report.  ( 
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1.  updating  the  projections  of  demand  for  trained  educational 
research,  development,  diffusion,  and  evaluation  (RDDE) 
personnel  in  1974,  based  on  the  current  economic  situation; 

2.  examining  the  possibility  that  there  were  significant  dis- 
crepancies between  number  and  type  of  positions  listed  in 
the  budgets  of  funded  proposals  and  the  number  and  type  of 
persons  who  were  actually  employed  to  carry  out  the  projects 
or  programs. 

Significance  of  the  StudV*^ 

Trainers,  training  program  administrators,  and  students  who  are 
prospective  educational  RDDE  professionals  have  a  very  practical  need 
to  know  the  range  of  possibilities  with  respect  to  the  future 
demand  for  and  employment  of  educational  RDDE  personnel.   The  necessity 
for  having  data  to  support  the  sensible  allocation  of  human  resources 
appears  to  be  justification  enough  for  carrying  out  such  a  study. 

The  justificatipn  need  not  be  quite  so  narrow,  however.  Policy 
makers  and  manpower  specialists  generally  have  taken  the  position  that 
the  trend  of  events  in  this  country  is  leading  toward  an  expanded  demand 
for  persons  who  can  help  to  improve  the  quality  of  life  in  the  United 
States.    For  example,  the  authors  of  a  recent  manpower  study  concluded 
the  rate  of  production  of  Ph.D.'s  should  be  reduced,  but  only  in  those 
fields  where  there  was  an  oversupply,  e.g.,  the  mathematical  sciences, 
the  physical  sciences,  and  engineering.    The  supply  of  trained  persons 
in  the  life  sciences  and  applied  social  sciences  was  said  to  be  "criti- 
cally short,''  and  the  autHjjjlreconinend 
fVs^j^^^^hese  areas  be  increased. '(dean  v^  rt 
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More  directly,  in  a  survey  supported  by  Phi  Delta  Kappa,  the  deans 
of  the  doctorate-producing  schools  of  education  in  the  country  responded 
that  "the  most  critically  needed  program  expansion(s)  or  program  develop- 
ment(s)  in  the  field  of  education  at  this  time"  was  "educational  research" 
(Robertson  and  Sistler,  1971,  p.  63).^ 

The  authors  of  the  PDK  study  note  their  belief  that  the  deans  were 
not  necessarily  indicating  that  expansion  of  conventional  educational 
research  programs  was  needed.    Instead,  the  authors  believe  the  adminis- 
trators were  sa^ying  expanded  training  was  most  critically  needed  in  the 
areas    commonly  referred  to  as  educational  research  and  evaluation  and 
development  and  diffusion. 

When  the  range  and  variety  of  alternatives  to  our  present  public 
schools— which  are  alreadly  being  tested— are  considered  (e.g.,  voucher 
systems, performance  contracts,  freedom  schools,  community-controlled 
schools),  together  with  the  pressures  which  are  being  brought  more 
heavily  to  bear  on  schools  (e.g. ,  integration,  drug  and  sex  education, 
economic  and  educational  accountability),  it  is  easy  to  agree  with  the 
deans  that  in  the  next  several  years  the  schools  will  most  need  the  help 
of  a  corps  of  "field  engineers,"  so  to  speak.    Educational  RDDE  personnel 
are  viewed  here  as  fulfilling  that  function.    It  goes  without  saying 
that  the  effectiveness  of  these  educational  RDDE  personnel  will  be 
contingent  upon  their  training— and  the  training  to  work  in  such  a 
.  conplex  milieu  takes  time.   That  time  will  be  available  only  if  there  is 
advance  planning  and  programming.   This  study  will  serve  as  an  aid  to 
those  who  must  do  the  planning  and  programming. 


:  ^Three  other  areas  named  ri6ar  the  top  of  the  list  were  urban 
education,  early  Chi I  dhpod  education,  and  junior  and  community  coll ege 
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Limitations  of  the  Study 


From  the  outset,  this  stucly  had  two  very  specific  objectives: 
(1)  to  update  (not  replicate)  the  projections  of  demand  made  in  the 
original  study  by  Clark  and  Hopkins,  and  (2)  to  test  the  validity  of  the 
assumption  that  the  number  and  type  of  positions  listed  in  the  budgets 
of  approved  proposals  would  lead  to  the  employment  of  that  number  and 
type  of  personnel . 

These  objectives  imposed  a  series  of  limitations  on  the  study: 

1.  Without  a  re-examination  of  the  basic  data  sources,  any 
updated  projections  of  the  t^pe  of  positions  available  in  1974  would 
necessarily  be  limited  to  research,  development,  and  diffusion  positions. 
Rather  than  tend  to  perpetuate  the  impression  that  evaluation  and 
evaluators  do  not  compose  an  important  segment  of  the  educational  RDDE 
community,  it  was  decided  not  to  prepare  a  projection  of  the  types  of 
position  to  be  available  in  1974.^ 

2.  No  attention  could  be  given  to  programs  which  had  been 
initiated  since  the  .data  for  the  original  study  were  gathered  in  May 
1966.   Hence,  nothing  could  be  included  here  on  such  .efforts  of  the 
Office  of  Education  as  the  experimental  schools  project,  the  Anacosftia 
project,  the  various  undertakings  in  reading,  and  so  forth.   On  the 


.  ,  ...................  .     .     .      .  ■  . 

^[Editor's  note]   It  is  not  possible,  With  the  data  gathered  for 
this  stu<<y,  to  differentiate  among  the  functions  of  RDDE.    If  one  is  will- 
ing to  accept  the  proportions  projected  in  the  original  study,  however, 
as  still  being  the  likely  trend,  then  those  proportions  (research:  33%, 
development:   50%,    diffusion:   17X)  may  be  applied  to  the  projected  total 
number  of  positions  presented  later  in  this  chapter  (Clark  and  Hopkins, 
1969,  p.  288). 
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other  hand,  on  the  surface  It  does  not  appear  that  these  programs  will 
einploy  such  large  numbers  of  RDDE  personnel  that  the  failure  to  include 
them  seriously  compromises  the  projection  of  the  present  study. 

3.  It  was  not  possible  to  examine  recent  proposals  to  determine 
whether  there  had  been  a  change  in  the  number  or  type  of  persons  being 
supported  by  the  various  programs;  consequently,  the  original  ratios  of 
people  to  dollars  expended  were  used  In  updating  the  projections. 

4.  There  was  no  •possibility  of  examining  the  means  by  which  the 
demand  for  personnel  In  FY  '66-'68  was  satisfied,  or  the  means  by  which 
the  Initial  and/or  In-service  training  of  the  people  who  moved  Into  the 
field  at  that  time  was  conducted. 

In  addition,  events  an^d  circumstances  have  created  another  set 
of  conditions  which  has  produced  limitations  In  the  present  study.  These 
are: 

1.   There  are  now  so  many  unknowns  in  the  program  administrators' 
environment  that,  for  the  most  part,  they  were  either  unable  or  unwilling 
to  project  the  course  of  their  programs  even  a  few  years  Into  the  future. 
Their  uncertainties  about  the  effect  of  a  National  Institute  for  Educa-  . 
tion  (If  there  Is  one),  the  health  of  the  national  econoniy,  national 
priorities,  and  similar  matters  have  produced  a  situation  where  only 
two  of  the  administrators  Interviewed  would  state  in  specific  terms 
Where  he  was  projecting  his  program  to  be  by  1974.   All  of  the  others 
used^such  non-specific  statements  as  "The  program  probably  w^n't  grow 
very  much."  - 

This  was  a  remarkable  change  from  1966,  when  many  of  the  same 
administrators  were  willing  to  project  eight  yeiirs  into, the  future  rather 
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than  two  or  three  years.   One  effect  of  the  administrators'  unwillingness 

to  be  speciVic  is  that  there  is  no  basis  for  preparing  a  multiple  set 

of  projections  which  encompaiss  the  likely  range  of  possibilities. 

Therefore,  unlike  the  original  study  (where  there  were  three  projections: 

Most  Optimistic,  Least  Optimistic,  and  Most  Likely),  just  one  projection 
5 

has  been  produced.     Another  effect  of  their  non-specificity  is  that  the 
present  writer  has  had  to  assign  many  more  values  arbitrarily  than  was 
true  in  the  original  stu<|y.    These  arbitrary  assignments  will  be  identi- 
fied. 

2.     Some  of  the  programs  examined  in  1966  have  since  modified 
their  direction,  emphases,  implementation  vehicles,  and  sources  of 
control.   Without  an  examination  of  recent  proposals,  it  was  necessary 
to  use  the  original  data  to  make  overall  projections,  but  the  program 
changes  made  it  seem  unwise  to  use  the  original  data  to  prepare  refined 
projections  of  the  spread  of  positions  across  the  range  of  institutional 
settings. 

Some  of  the  funds  included  in  the  original  study  have  since 
been  re-allocated  to  support  programs  not  included  here  (e.g.,  $12 
million 'for  the  support  of  the  experimental  scftopls  project).^  Since 


It  Is  likely  there  would  not  have  been  much  spread  if  multiple 
projections  had  been  prepared  because  FY  '74  is  now  so  near  that  the 
opportunity  for  divergence,  is  greatly  reduced. 

^It  would  not  be  accurate  to  say  the  experimental  school  monies 
"came  out  of"  the  funds  used  to  support  project  R&D,  for  example,  because 
there  is  no  carry-over  commitment  to  any  program  from  one  year  to  the 
next.   However,  there  was  a  distinct  decline  in  the  funds  made  available 
for  support  of  project  R&D  when  the  funds  were  awarded  .to  start  the 
experirnental  schools  project. 
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the  new  programs  are  not  included  here,  the  projections  of  the  original 
stucly  and  this  updated  version  are  based  on  differing  dollar  bases  hence 
the  results  of  the  two  studies  are  not  entirely  comparable.   The  updated 
projections  rest  on  a  dollar  base  which  is  approximately  four- fifths  the 
size  of  the  base  for  the  original  projections.    However,  this  smaller 
dollar  base  seems  more  defensible. 

4.     There  is  little  doubt  that  the  heavy  infusions  of  funds  from 
the  ESEA  of  1965  produced  an  enthusiasm  which  caused  the  original  projec- 
tions to  be  unduly  optimistic.    However,  the  extended  financial  starvation 
of  the  R  and  D  programs  since  that  time  appears  to  have  caused  the  program 
planners  now  to  become  unduly  pessimistic  in  their  views  of  the  future. 
This  pessimism  is  reflected  in  the  projection,  since  it  represents  the 
views  of  the  men  who  run  the  programs,  but  the  reader  is  alerted  that 
this  may  rt;sult  in  the  projection  being  somewhat  lower  than  is  warranted. 

Description  of  the  Methodology 

The  projection.    A  technical  discussion  of  the  projection  metho- 
dology can  be  found  in  chapter  three  of  the  Clark  and  Hopkins  report. 
The  procedui^s  followed  to  update  the  original  projections  were  as 
follows : 

1.     Program  administrators  were  interviewed  to  obtain  (a)  data 
on  expend-ftures  for  the  period  FY  '69-'71,  (b)  information  on  their 
FY  '72  allocations,  and  (c)  projections  of  the  growth  of  their  programs 
during  FY  '73  and  '74.    Information  was  also  sought  on  the  progress  of 
the  various  sub-units  in  their  programs  which  were  used  as  part  of  the 
original  projections  (e.g.,  the  number  and  growth  of  R  and  D  centers, 
the  progress  of  the  educational  laboratory  program,  the  phase-out 
schedule  for  discontinued  units). 
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2.  The  financial  and  program  information  received  was  broken 
into  the  same  sub-units  as  were  used  in  the  original  study.  [There 
was  one  exception,  which  will  be  discussed  In  Table  20.] 

3.  As  in  the  original  study,  the  prf.-jected  FY  '74  amount  was 
compared  with  the  equivalent  of  the  FY  '68  dollars  and  an  indicator  of 
true  growth  obtained.^    (See  Table  55  on  pp.  229-230  of  the  Clark- 
Hopkins  report.)    For  example,  the  FY  '74  equivalent  of  the  FY  '68 
funding  base  of  $1  million,  after  five  percent  per  year  is  added  to 
offset  increases  in  costs,  is  $1  ,477,000.    A  program  with  just  such  a 
funding  history  would  be  assigned  a  "growth  ratio"  of  1.0. 

4.  The  growth  ratio  was  then  applied  to  the  base  nuinber  of 

positions  available  in  each  of  the  sub-units  (e.g.,  in  the  R  and  D 

centers,  in  the  ERIC  clearing  houses,  etc.).    These  base  numbers  are 

presented  in  Table  56  of  the  original  study  (p.  237).    This  produced  the 

number  of  positions  the  projected  FY  '74  funds  would  support,  assuming 

8 

it  would  take  about  the  same  number  of  equivalent  dollars   to  support 
each  position. 


A  five  percent  per  year  (to  FY  '74)  increase  was  applied  to  all 
funding  figures  to  provide  "no  real  gain"  figures.    This  "no  real  gain" 
projection  was  made  "in  order  to  determine  the  equivalent  funding  base 
needed  in  FY  '74  to  support  the  same  number  of  people  as  were  supported 
in  FY  '66"  (Clark  and  Hopkins,  1969,  p.  140). 

o 

"The  term  "equivalent  dollars"  is  used  to  refer  to  the  five  percent 
per  year  increase  in  funds  from  FY  '68  to  FY  '74.    In  the  example  given 
in  No.  3  above,  the  $1  ,477,000  in  FY  '74  are  "equivalent  dollars"  to 
the  $1 ,000,000  in  FY  '68. 
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5.     The  total  number  of  positions  projected  to  be  supported  by 
USOE  and  the  NSF  course  content  improvement  programs  in  FY  '74  was  con- 
sidered to  be  a  baseline  number  of  RDDE  positions  to  be  available.  There 
were  entire  populations  of  positions  not  yet  represented  (e.g.,  schools 
of  education,  business  and  industry).    The  next  step  was  to  go  through  the 
original  studiy  and  itemize  each  of  these  populations  and  add  them,  or  an 
appropriate  proportion  of  them,  to  the  baseline  projection.    This  produced 
the  final  projection  of  the  study. 

The  follow-up  study  of  persons  employed.    Follow-up  information  on 
actual  employment  was  sought  from  directors  of  a  sample  of  the  projects 
and  programs  Included  in  the  original  stu4y.    The  procedures  followed 
were  the  following: 

1.  A  list  was  compiled  of  the  numbers  and  institutional  loca- 
tions of  all  of  the  projects  included  in  the  original  Clark  and  Hopkins 
stucjy. 

2.  A  search  was  made  of  ERIC  and  the  NCERD  budget  office's 
records  for  the  hame(s)  of  the  princial  investigator(s)  of  each  project. 

3.  ffembership  directories  (e.g.,  APA,  AERA)  and  the  National 
Faculty  Directory  were  searched  for  current  addresses  for  the  investiga- 
tors listed. 

4.  The  projects  were  then  arranged  by  USOE  support  programs 
(e.g..  Higher  Education  Research,  Elementary  and  Secondary.  Research) 
and,  using  a  table  of  random  numbers,  a  20  percent  sample  of  projects 
was  selected  from  each  program  (with  the  one  stipulation  that  no  ^program 
was  represented  by  fewer  than  five  projects). 
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5.  An  initial  letter  of  inquiry  was  sent  to  investigators  May  28, 
1971;  a  follow-up  letter  sent  to  non-respondents  on  June  25;  and,  finally, 
a  postcard  sent  to  those  who  had  still  not  responded  as  of  July  14. 

6.  The  responses  received  were  then  separated  into  (a)  projects 
where  there  had  been  changes  in  the  persons  employed  from  the  persons 
listed  in  the  budget  of  the  original  proposal,  and  (b)  projects  in  which 
there  had  been  no  such  changes. 

7.  The  projects  in  which  there  had  been  changes  were  then  fur- 
ther separated  according  to  whether  the  changes  appeared  to  have  or  not 
have  significance  for  the  original  and  current  manpower  projections. 
The  analysis  simply  reports  the  degree  and  level  of  the  changes  reported 
by  the  principal  investigators. 

Presentation  of  the  Data 

The  point  was  iuade  in  the  original  Clark  and  Hopkins  study  that 
the  reader  should  focus  chiefly  on  the  magnitude  of  the  difference  in 
the  positions  available  over  time,  rather  than  upon  the  specific  number 
of  positions  said  to  be  available  or  the  precise  difference.    That  point 
bears  repeating  here.   The  projected  numbers  are  imaginary.    They  are 
used  only  to  indicate  to  program  pi anners ,  administrators,  and  directors 
the  direction  and  scope  for  which  they  should  develop  contingency  plans- 

What  follows  is  a  projection  of  the  positions  which  could  be 
supported  if  the  anticipated  funds  are,  in  fact,  made  available  and  the 
anticipated  developments  do  occur.    It  is  not  a  prediction  of  the  1974 
situation;  neither  is  it  a  forecast.    The  value  of  the  original  projec- 
tions was  that  a  concerned  individual  could  readily  check  the  developments 
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which  occurred  after  the  projections  were  developed  and  could  then 
revise  them  accordingly--in  a  much  shorter  period  of  time  than  it  would 
have  taken  to  develop  a  current  projection  from  scratch.    The  present 
projection  is  a  demonstration  of  the  utility  of  the  methods  used. 
Obviously,  however,  the  current  projection  is  not  as  extensive  as  were 
the  projections  in  the  original  study.    The  new  projection  is  sinply  and 
solely  an  update,  based  on  the  best  evidence  obtainable,  for  the  benefit 
of  persons  interested  in  the  approximate  magnitude  of  growth  of  the 
educational  RDDE  community. 

Baseline  projection.   The  baseline  projection  of  this  study,  i.e., 
the  projection  of  positions  to  be  supported  in  FY  '74  by  selected  USOE 
and  NSF  funding  programs,  is  presented  in  Table  20.  Since  the  baseline 
projection  fails  to  account  for  large  numbers  of  positions  winch  have 
other  sources  of  support  (e.g.,  state  and  local  governments,  public 
schools,  universities),  additional  populations  are  added  later  to  get 
the  final  projection  of  the  study  with  respect  to  the  number  of  positions 
to  be  available  in  educational  RDDE  in  FY  '74. 

The  FY  '74  projected  funding  figures  were  derived  as  follows: 
1.     R  and  D  centers,  early  childhood  centers,  policy  study 
centers,  and  laboratories— the  program  administrator  furnished  generalized 
projections  into  FY  '73;  the  writer  arbitrarily  assigned  a  value  for 
these  programs  in  FY  '74,  using  the  funding  history  of  these  programs 
FY  '66-'71  and  the  program  administrator's  FY  '72  and  '73  projections 
as  guides. 
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2.  Vocational  education  R  and  D  centers— according  to  a  member 
of  the  program  staff,  these  centers  are  to  be  raised  to  their  full  funding 
level  of  $3.3  million  (combined)  in  FY  72-'73  and  held  there,  with  only 
tiny  additions  of  funds  to  offset  inflationary  inroads.   The  writer 
arbitrarily  added  $50,000  for  FY  '74,  as  a  "tiny  addition." 

3.  Handicapped  children  and  youth  R  and  D  centers  and  materials 
centers—the  projected  figures  are  the  administrator's  projections. 

4.  ERIC  clearing  houses—the  program  administrator  furnished 
generalized  projections  into  FY  '72;  the  writer  extended  his  projections 
into  FY  '74  using  the  same  guides  as  indicated  in  the  first  category 
above. 

5.  Vocational  education  research  coordinating  units  (RCU's) — 
funds  for  the  RCU's  are  no  longer  administered  by  the  USOE,  but  are  sent 
directly  to  the  states  where  they  are  used  to  support  traini ng  programs, 
R  and  D,  and  a  wide  variety  of  other  activities.    In  FY: '69  the  states 
received  approximately  $2,270,000  from  USOE  for  support  of  the  RCU's. 

In  FY  '71,  the  Congress  granted  the  states  $17,874,000  for  RCU  support. 
No  one  was  willing  (or  able)  to  project  (a)  what  Congress  would  decide 
in  the  future  regarding  RCU  support,  or  (b)  what  proportion  of  the  funds 
granted  would  be  used  by  the  states  to  support  RDDE  positions.  After 
considering  the  availability  of  qualified  personnel  and  the  behavior  of 
RCU  administrators  in  the  past,  one  staff  member  very  familiar  with  the 
program  estimated  a  10  percent  expansion  in  the  number  of  RDDE  positions 
which  could  be  supported  to  FY  '74.    The  writer  used  this  projection  to 
assign  a  value  of  $2,500,000  of  RCU  funds  to  be  used  in  FY  '74  for  educa- 
tional RDDE  as  defined  in  the  original  study. 
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6.  State  department  research  units—funds  for  this  program  have 
not  increased  in  five  years,  although  the  proportion  over  which  USOE  has 
exercised  control  has  fluctuated.    In  view  of  this  funding  history,  and 
the  program  administrator's  (or  anyone  else's)  inability  to  project  the 
mind  and  will  of  Congress,  the  writer  simply  extended  to  FY  '74  the 
same  $28,262,000  the  program  has  received  in  the  past,  and  assigned 

10  percent  of  that  amount  to  the  support  of  RDDE  positions,  as  explained 
in  the  original  stuc(y. 

7.  Small  projects— the  FY  '74  figure  is  the  projection  of  the 
administrator  of  this  program. 

8.  Regular  projects— funds  are  combined  in  this  category  from 
the  handicapped  children  and  youth  program  and  the  National  Center  for 
Educational  Research  and  Development,  USOE.   The  handicapped  children  and 
youth  program  portion  of  the  total  is  based  on  a  firm  estimate  of  FY  '74 
funding  by  the  program  administrator.   The  NCERD  portion  of  the  FY  '74 
total  is  based  on  hard  and  detailed  information  from  the  program  adminis- 
trator through  FY  '72,  and  an  extension  of  this  information  into  FY  '74  by 
the  present  writer  using  the  guidelines  stated  in  the  first  category  above. 

9.  Title  III  centers— the  funding  history  of  this  program  has 
been  as  follows: 


FY  '68 


$209,000,000 


FY  '69 


164,876,000 


FY  '70 


116,693,000 


FY  '71 


143,243,000 


FY  '72 


146,248,000  (est.) 
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As  was  the  case  with  the  administrator  of  the  state  depart- 
ment program  (category  No.  6  above),  this  administrator  was  simply  unable 
to  project  the  mind  and  will  of  Congress.    The  funding  history  offered 
no  consistent  pattern,  so  the  writer  did  not  attempt  to  tread  where  the 
program  administrator  feared  to  step.    The  FY  '74  projection  for  this 
program  represents  six  percent  (as  in  the  original  stucly)  of  a  flat 
continuation  of  the  program  administrator's  estimate  of  FY  '72  funding. 

10.     NSF  course  content  improvement  program--the  funding  history 
of  this  program  has  been  as  follows: 


FY  '68  -  $  13,500,000 

FY  '69  -  7,975,000 

FY  '70  -  6,200,000 

FY  '71  -  5,800,000 

FY  '72  -  5,800,000  (est.) 


The  administrator  of  this  program  thought  the  funding  curve 
would  "bottom  out"  in  FY  '71  and  '72,  and  start  a  gradual  clini)  there- 
after.   Based  on  his  generalized  projections,  the  writer  arbitrarily 
assigned  the  value  of  $6,500,000  in  FY  '74  for  this  program. 

Final  projection.    The  baseline  projection  fails  to  account  for 
populations  of  RODE  positions  which  are  not  supported  by  the  selected 
USOE  or  NSF  programs  included  in  the  stucly.   The  populations,  identified 
in  the  original  studly  as  the  components  of  the  1964  community,  are 
examined  in  the  paragraphs  which  follow. 

1.     School  and  college  of  education  populations.    The  original 
studly  added  to  tr»e  basline  projections      the  160  persons  employed  in 
research  bureaus  in  1964;  the  44  education  personnel  who  were  employed 
by  state  departments  to  staff  RCU's;  the  40  research  stimulator- 

"  2:1 
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coordinators  who  were  employed  by  doctoral -granting  institutions;  and 


engaging  in  project  activity. 

In  the  absence  of  any  substantial  expansion  of  the  funds 
available  through  the  major  USOE  and  NSF  support  programs  included  in 
the  study,  there  is  no  basis  for  any  projection  beyond  the  number  of 
persons  who  were  alreadly  employed  by  other  sources.    There  may  actually 
be  an  increased  number  of  individuals  working  on  educational  RDDE  projects, 
but  there  is  nothing  in  the  data  developed  for  this  study  which  would 
indicate  that  is  the  case.    Consequently,  the  writers  did  not  project 
beyond  the  1964  data.    Thus,  only  the  1  ,000  individual  project  personnel 
identified  in  1964  were  included.   The  present  study  retained  from  the 
above  populations  only  those  positions  which  were  supported  by  other 
sources  in  1964,  to  wit: 


2.     The  positions  identified  in  academic  settings  other  than 
schools  and  colleges  of  education  in  1964  so  closely  resembled  each 
other  that  they  can  be  presented  in  tabular  form.    All  of  these  positions 
were  added  to  the  baseline  projection,  for  the  same  reasons  as  were 
given  in  the  discussion  of  school  and  college  of  education  positions. 


2,261  individuals  (in  the  Least  Optimistic  projection)  who  would  be 


160 
44 
40 
1.000 


personnel  in  research  bureaus 
personnel  employed  to  staff  RCU's 
stimulator-coordinators 
individual  project  personnel 


1,244 


total  in  schools  and  colleges 
of  education 


Setting 


Bureau  Individual 
Personnel    Project  Personnel 


Schools/departments  of  psychology  70 

Other  behavioral/social  science  depts.  64 

Other  discipline/academic  depts.  20 

Col  lege/ university  administration  150 


430 
463 
471 
55 


uni  ts 


TOTAL 


1 ,419 
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It  Should  be  rointed  out  that  in  categories  3-9,  the  numbers 
given  are  those  for  the  number  of  RDDE  personnel  employed  in  the  particu- 
lar agency  and  not  the  total  nurriber  of  personnel  employed  in  an  RDDE 
agency. 

3.  U.  S.  Office  of  Education.    There  continue  to  be  no  better 
data  available  than  the  1966  data  cited  in  the  original  study.  These 
indicated  there  were  approximately  136  program  directors  and  professional 
staff  personnel  in  RDDE  units  within  USOE  and  about  20  stimulators- 
coordinators  of  RDDE.    All  of  these  persons  are  engaged  in  facilitation 
or  management  of  RDDE.    All  156  positions  were  added  to  the  baseline 
projection. 

4.  State  departments  of  education.   The  states  were  supporting 
about  240  studjy,  planning,  evaluation,  and  resaarch  positions  in  1964. 
They  also  supported  92  RCU  positions  not  previously  accc-mted  for.  There 
were  115  individual  project  people  and  10  stimulators  and  coordinators. 
All  were  added  to  the  baseline  projection. 

5.  Schools  and  school  systems.    Research  bureaus  of  school 
systems  employed  about  265  persons  in  1964.    Another  five  stimulators 
and  coordinators  were  identified.    All  270  were  added  to  the  baseline 
projection. 

6.  Private  research  institutes  and  agencies.    No  change  was 
made  in  the  300  positions  being  supported  in  this  setting  in  1964;  all 
were  added  to  the  baseline  projection. 

7.  Professional  associations.    The  90  program  personnel  reported 
in  this  setting  in  1964  were  added  to  the  baseline  projection. 
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8.  Inter-agency  organizations.    There  were  50  persons  employed 
in  1964  by  such  consortia  as  the  Association  of  Great  Lake".  Colleges  and 
the  Compact  of  the  States.   This  number  has  undoubtedly  grown,  but 

in  the  absence  of  any  basis  for  estimating  the  extent  of  that  growth, 
just  50  positions  were  added  to  the  baseline  projection. 

9.  Business  and  industrial  organizations.   This  setting  was  not 
adequately  represented  in  either  the  1964  description  or  the  baseline 
projections  of  the  original  study.   One  hundred  fifty  positions  were 
identified.    It  is  true  that  the  "education  industry"  has  not  grown  to 
nearly  the  extent  projected,  but  that  it  has  .expanded  is  obvious  from  the 
number  of  new  corporations  which  have  been  formed  since  1964.  Rather 

than  limit  the  positions  in  this  setting  to  the  nuiriber  inadequately  identi- 
fied in  1964,  the  writer  has  arbitrarily  doubled  it,  asserting  that  there 
will  be  (if  there  are  not  already)  at  least  300  educational  RDDE  positions 
in  business  and  industrial  organizations  throughout  the  country  in  1974. 
In  this  setting,  it  is  recognized,  the  writer  is  making  a  prediction 
rather  than  a  projection. 

To  summarize,  the  following  populations  of  educational  RDDE  per- 
sonnel were  identified  as  having  been  in  existence  for  a  period  of  time 
and  likely  to  continue  in  existence  without  reference  to  the  vagaries  of 
federal  funding: 


Setting 


Popul  ation 


Schools  and  colleges  of  education 
Schools  and  departments  of  psychology 
Other  behavioral  and  social  science  depts. 
Other  discipline  and  academic  depts. 
College  and  university  administrative  units 


1,244 
500 
527 
491 
205 
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Setting 


Population 


U.  S.  Office  of  Education 
State  departments  of  education 
Schools  and  school  systems 
Private  research  institutes  and  agencies 


Professional  associations 
Inter-agency  organizations 


156 
457 
270 
300 
90 
50 
300 


Business  and  industrial  organizations 


TOTAL 


4,590 


When  the  baseline  projection  of  positions  to  be  supported  by 
selected  USOE  and  NSF  programs  is  added,  we  have  the  following  final 
projection. 


For  comparative  purposes,  the  reader  will  find  it  of  interest  that 
the  original  Clark  and  Hopkins  study  Identified  4,125  persons  in  the  1964 
educational  RDDE  community.    Most  of  the  growth  which  is  indicated  by 
the  distance  between  the  1964  and  1974  totals  undoubtedly  took  place 
during  the  short  span  in  1965  and  1966  when  the  new  ESEA  agencies  and 
centers  were  staffing  up.    Although  we  do  not  have  data  to  indicate  the 
number  of  persons  actually  employed  in  RDDE  positions  at  the  present  time, 
it  seems  reasonable  to  assume  that  most  of  the  8,669  projected  to  be 
needed  by  1974  ar«  already  employed.   The  reasons  for  this  assumption 


Baseline  projection 

Positions  supported  from  other  sources 

Final  projection  of  positions 


4,079 
4.590 


8,669 


research:  2,861 
development:  4,334 
diffusion:  1,474 
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are  that  the  economic  indicators,  at  least,  suggest  that  since  1966 
there  has  been  little  expansion  in  the  number  of  educational  RDDE  posi- 
tions available— nor  are  funding  program  administrators  projecting  that 
there  will  be  any  extension  in  the  capacity  to  support  such  positions 
in  the  near  future. 

Follow-up  Study  of  Persons  Employed 

Questionnaires  were  sent  to  46  project  investigators  in  an  attempt 
to  learn  whether  the  positions  listed  in  the  budgets  of  proposals  approved 
by  USOE  were  actually  filled— as  assumed  by  Clark  and  Hopkins.    The  ques- 
tionnaires were  distributed  to  a  random  sample  of  project  directors  in 
each  of  the  USOE  departments  and  divisions  in  the  original  study  which 
supported  project  activity  (as  opposed  to  laboratories,  centers,  etc.). 
The  number  of  questionnaires  sent  and  usable  returns  received,  by  USOE 
program,  were: 


USOE  Program 

Sent 

Recei  ved 

Higher  Education  Research 

15 

11 

Adult  and  Vocational  Research 

10 

10 

ERIC 

5 

4 

Elementary/Secondary  Research 

6 

4 

El  ementary/Secondary  Curri  cul  um 

5 

4 

Handicapped  Children/Youth 

_5 

_4 

TOTAL 

46 

37  (80%) 

Five  of  the  questionnaires  were  returned  with  notes  indicating 
they  had  been  sent  to  the  wrong  person  or  could  not  be  delivered.  Four 
were  simply  not  returned. 

Of  the  37  usable  returns  received,  23  (62  percent)  reported  no 
difference  of  any  kind  between  the  positions  listed  in  the  budgets  sub- 
mitted for  approval  and  the  personnel  actually  employed  during  the  dura- 
tion of  the  project. 
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A  difference  between  the  proposed  project  staff  and  the  staff 
actually  employed  was  reported  by  14  investigators.    The  changes  reported 
are  listed  below,  grouped  by  the  writer  into  two  categories:    (1)  eight 
cases  which  could  possibly  affect  the  Clark  and  Hopkins  manpower  proi^^ 
tion  significantly,  and  (2)  six  cases  which  appeared  unlikely  to  affjPt 
the  manpower  projection  in  any  significant  viay. 

Possibly  significant  changes  in  staffing 

1.  Used  two  fewer  programmers  than  anticipated;  also 
replaced  two  graduate  assistants  in  education  with  a  graduate 
student  who  had  training  in  mathematical  logic—which  was  needed 
in  t!;e  stu4y. 

2.  Added  one  programmer. 

3.  Added  two  writers  and  one  visual  artist. 

4.  Employed  ^vo  research  associates  instead  of  a 
programmer. 

5.  Personnel  desired  were  not  available  for  employment. 

6.  A  co-investigator  was  added;  also  a  one-third  time 
consultant. 

7.  One  fewer  research  associate  was  employed;  two  addi- 
tional graduate  assistants  were  hired  instead. 

8.  Did  not  employ  one  special  coordinator,  two  researdi 
design  specialists,  one  publications  director,  or  six  graduate 
student  interns  because  they  were  not  needed  (these  would  have 
been  short-term,  part-time  employees  if  they  had  been  called  upon). 
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Probably  insignificant  changes  in  staffing 

9.     Used  10  field  workers  for  a  brief  period  instead  of 
two  field  workers  for  a  longer  period,  because  of  shortened 
deadlines. 

10.  Three  interviewers  added  (paid  for  each  interview 
completed). 

11.  Employed  four  research  assistants  instead  of  four 
graduate  assistants. 

12.  Two  co-investigators  worked  one-half  tinie  on  the 
project,  rather  than  full-time  as  scheduled. 

13.  Added  one  half-time  observer  and  two  test  scorers. 

14.  Employed  14  graduate  students  at  an  hoijrly  rate. 

The  latter  reports  are  termed  "probably  insignificant"  because  they 
appear  to  Involve  (1)  para-professionals  who  would  not  have  been  included 
in  the  original  stu4y  anyws^,  (2)  persons  who  were  probably  employed 
less  than  the  equivalent  of  20  percent  time  for  a  year— which  was  the 
cut-off  point  used  in  the  original  study ^  and  (3)  differences  in  time 
or  title  rather  than  persons  employed. 

It  Is  difficult  to  sa^y  what  significance  the  first  eight  cases 
have  for  the  manpower  projection  base.   The  additions  and  subtractions 
seem  to  fall  into  a  rough  balance,  but  it  is  recognized  this  could  be 
said  to  be  adding  apples  and  oranges.   There  is  nothing  reported  there 
(or  anywhere  In  this  section)  which  appears  to  the  author  to  raise 
serious  doubts  about  the  validity  of  the  assumption  by  Clark  and  Hopkins 
that  the  positions  listed  In  budgets  approved  by  the  USOE  did  represent 
positions  which  would  be  filled  by  the  employment  of  educational  RODE 
persons.    In  fact,  the  returns  appear  to  substantiate  the  assumption. 
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Conclusions 

!•    The  projections  of  the  original  Clark  and  Hopkins  study 
indicated  the  1964  population  of  4,125  RDDE  personnel  would  by  1974 
increase  at  least  three-fold,  possibly  seven-fold,  but  most  likely  five- 
fold.   All  three  projections  now  appear  to  be  overly  optimistic.  Accord- 
ing to  the  data  compiled  for  the  present  study,  there  will  more  likely  be 
a  doubling  of  the  1964  population  by  1974,  to  approximately  8,669  persons, 
assuming  all  positions  are  filled. 

2.  The  projection  in  this  paper  may  be  too  pessimistic.    It  is 
likely  there  has  been  some  expansion  since  1964  among  university  person- 
nel ,  state  department  RDDE  personnel,  private  research  institute  personnel, 
and  personnel  in  other  settings,  even  with  the  slowdown  in  government 
support.   And,  in  spite  of  the  fiscal  history  for  educational  RDDE  since 

FY  '68,  flat  funding  of  federal  programs  is  decidedly  unusual,  hence  it 
seems  unlikely  that  there  will  not  be  some  expansion  in  federal  support 
of  educational  RDDE  between  the  present  and  FY  '74.    To  repeat,  however, 
the  projection  in  this  study  does  not  take  into  account  either  of  these 
factors,  because  there  are  no  data  available  to  support  any  such  exten- 
sion. 

3.  According  to  the  information  on  economic  indicators  obtained 
from  interviews  with  program  administrators,  there  will  be  no  upturn  in 
the  demand  for  educational  RDDE  personnel  before  FY  '^. 

4.  It  was  valid  to  assume  that  the  positions  listed  in  the 
budgets  of  projects  approved  for  federal  funding  in  FY  '65  accurately 
reflected  jobs  w^'ik  would  be  opened  up  to  educational  RDDE  personnel. 
Both   the  original  and  the  present  projections  rest  on  this  assumption. 
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Recommendations 

1.  If  the  third  oonclusion  above  is  valid,  then  training  pro- 
gram directors  should  counsel  their  students  that  there  is  no  indication 
.there  will  be  any  sizeable  nuirber  of  unoccupied  positions  in  educational 
RDDE  between  now  and  1974. 

2.  Administrators  of  training  support  programs  or  possibly  AERA 
should  take  note  of  the  group  of  approximately  4,000  persons  who  moved 
Into  the  educational  RDDE  field  in  1966-68,  and,  on  the  assunption  these 
persons  are  not  adequately  or  at  least  broadly  trained,  make  provision 
for  them  to  receive  some  advanced  training  while  on  the  job. 

3.  During  this  hiatus  In  demand  for  newly-trained  persons, 
administrators  of  training  support  programs,  AERA,  and  others  Interested 
In  training  should  take  advantage  of  the  opportunity  to  (a)  engage  in 
research  on  training,  (b)  re-examine  the  assumptions  about  training 
which  have  grown  out  of  custom  and  habit,  (c)  explore  possibilities  for 
creating  more  of  a  demand  for  educational  RDDE  In  settings  where  the 
demand  has  been  weak,  and  (d)  develop  models,  rationalized  systems,  and 
alternative  techniques  for  providing  more  efficient  and  effective  train- 
ing to  a  broader  variety  of  Individuals. 

4.  The  Office  of  Education »  AERA,  or  both  should  begin  now  to 
develop  a  mechanism  for  regularly  assessing  manpower  supply  and  demand 
In  RDDEy  and  then  use  that  mechanism  to  conduct  a  major  census  of  RDDE 
manpower  In  the  near  future.  Such  a  mechanism  might  borrow  techniques 
from  several  studies »  such  as: 

a.     Busweirs  study  of  education  doctorates  in  1954 
and  1964. 
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b.  Sieber's  stuc(y  of  research  units  in  1964. 

c.  The  National  Register  survey  in  1964. 

d.  The  Clark  and  Hopkins  description  of  the  1964 
community  and  their  projections  to  1974. 

In  addition,  the  possibility  exists  of  collecting  baseline  or  threshold 
data  on  the  age  of  the  National  Institute  for  Education,  as  the  above 
studies  did  for  the  age  of  ESEA-1965. 

A  new  data-based  effort  is  required,  in  any  case,  for  sensi- 
ble planning  because  all  of  the  above  studies  and  surveys  have  now  aged 
to  the  point  where  it  is  difficult  to  relate  them  to  present  circum- 
stances.  If  planning  were  begun  now  on  a  major  census  or  survey,  perhaps 
a  set  of  arrangements  could  be  worked  ojt  by  1974  or  1975  which  would 
permit  tlie  results  to  be  used  as  the  base  of  a  regularized  annual  or 
biennial  survey  of  educational  RDDE  manpower.    Until  there  is  a  nation- 
wide census  of  people,  training,  and  jobs,  estimates  of  the  congruence 
between  future  supply  and  future  demand  will  remain  more  an  art  than  a 
science. 

Observations 

1.     It  seems  reasonable  to  begin  training  people  in  advance  of 
a  major  new  thrust,  as  the  new  National  Institute  for  Education  certainly 
win  be.   That  has  not  been  the  practice  in  the  past,  however,  and  the 
projection  in  this  chapter  provides  no  basis  for  an  operating  level 
administrator  to  win  support  at  this  time  to  begin  training  to  meet  a 
future  need.    Experience  has  shown,  however,  that  there  must  be  training 
In  advance  of  future  needs;  the  large  nunters  of  inadequately  trained 
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persons  who  were  forced  into  RDDE  positions  which  arose  after  passage  of 
ESEA  is  testimony  to  that  belief.    Therefore,  it  appears  the  impetus 
for  training  personnel  to  meet  future  needs  will  have  to  come  from  the 
very  highest  levels  of  decision-making  in  the  Office  of  Education  and/or 
HEW.    Apparently  it  is  only  at  that  level  that  there  is  any  finn  view 
of  what  the  future  holds  for  educational  RDDE. 

2.     The  funding  reverses  of  the  past  several  years  have  caused 
more  harmful  effects  than  diminished  job  opportunities  or  cutbacks  in 
programs  and  projects.    Program  administrators  appear  to  be  so  uncertain 
(and  frustrated)  that  long-term  planning  is  being  neglected.    Bold  inno- 
vation has  been  replaced  with  maintenance  activity.    Enthusiasm  and  drive 
have  been  replaced  by  timidity.   One  must  wonder  what  other  costs  there 
have  been  and  will  be  as  a  result  of  this  induced  set  of  behaviors. 
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A  REVIEW  AND  SYNTHESIS  OF  STUDIES  ON  MANPOWER  SUPPLY  AND  DEMAiND  IN 
EDUCATIONAL  RESEARCH,  DEVELOPMENT,  DIFFUSION,  AND  EVALUATION 

Manpower  studies  involving  projections  of  future  demand  for  research, 
development,  diffusion  and  evaluation  (RDDE)  personnel  are  very  difficult 
to  conduct.   Projecting  a  demand  for  trained  personnel  is  complicated  even 
during  financially  stable  periods.   When  Federal  funding  practices  take  on 
many  Monte  Carlo  characteristics,  such  projections  become ^even  more  ten- 
uous.   Even  if  funding  problems  are  not  in  existence  (i.e.,  if  projections 
for  future  demand  are  made  regardless  of  the  financial  resources  available), 
manpower  studies  are  difficult  to  conduct  because  the  projections  are  usually 
based  (necessarily)  on  many  assumptions.    Hence,  the  projections  are  only  as 
valid  as  the  assumptions  underlying  them. 

Nevertheless,  In  the  past  few  years,  the  need  for  information  to 
guide  program  planners  has  led  several  members  of  the  educational  research 
community  to  attempt  to  determine  the  number  of  trained  persons  presently 
in  RDDE  positions,  the  number  of  persons  currently  being  prepared  to  fill 
existing  and  emerging  RDDE  positions,  and  the  number  of  persons  who  might 
be  needed  by  the  research  community  at  some  specified  future  date.    It  is 
the  purpose  of  this  chapter  to  describe  and  synthesize  those  studies 
relating  to  manpower  supply  and  demand.    Specifically,  the  chapter  contains 
information  relating  to  the  attainment  of  objective  5.2  of  the  current  Task 
Force  project,  which  was  originally  stated  as:    "To  synthesize  existing 
manpower  studies  based  on  nonflnancial  projections."   In  addition  to  non- 
financial  projections,  the  financially  based  projections  prepared  by  Clark 
and  Hopkins  (1969)  are  also  included  in  the  synthesis. 
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Before  the  studies  can  be  described,  it  is  necessary  to  explain 
the  process  by  which  the  studies  were  identified  and  chosen  for  inclusion 
in  this  report.    The  identification  of  studies  relating  to  manpower  supply 
and  demand  was  accomplished  as  part  of  the  work  leading  to  the  attainment 
of  Task  Force  Objective  6.    That  objective  was  to  identify  and  synthesize 
"extant  studies  of  educational  RDDE  personnel  or  studies  of  variables 
directly  related  to  such  persons."   The  reader  is  referred  to  Technical 
Paper  No.  27  in  the  Task  Force  series  for  a  detailed  description  of  the 
procedures  used  in  the  search,  description  and  judgment  phases  of  the  work  J 
A  brief  overview  of  those  procedures  will  be  given  here. 

Standard  library  search  methods  (e.g..  Psychological  Abstracts, 
Education  Index>  Research  Studies  in  Education,  ERIC)  were  used  to  identify 
all  works  which  nright  be  relevant  to  the  topic  of  researcVi  training.  Services 
of  the  Smithsonian  Institution's  Science  Information  Exchange  were  also  used 
to  identify  any  current  research  underway  in  the  area.   Approximately  275 
titles  were  thus  identified.   Of  these  2  75,  150  articles,  books  and  manu- 
scripts were  discarded  because  they  were  not  reports  of  data  based  studies 
or  because,  after  inspection,  it  was  determined  that  their  content  was  un- 
related to  RDDE  training.    The  remaining  125  studies  were  deemed  relevant  to 
Objective  6.   Of  these  125,  eight  studies  related  in  some  ws(y  to  the  focus 
of  this  paper. 

Those  studies  which  could  be  obtained  from  authors,  libraries  or  Uni- 
versity Microfilms  were  read  and,  if  they  met  the  inclusion  criteria  described 
in  Technical  Paper  No.  27^  they  were  described  in  some  detail  and  subjected 
to  tests  of  methodological  adequacy  by  moniirrs  of  the  AtRA  Task  Force  staff. 
The  description  outline  consisted  of  the  problem  statement,  the  statement  of 

Those  procedures  are  also  desgiiied  fully  in  the  first  section  of 
Chapter  VIII  of  this  report: 
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objectives,  and  the  hypotheses  or  questions  posed  in  the  study.  Data 
generation  was  described  according  to  the  population  studied,  the  sample, 
the  variables  studied,  and  the  instruments  or  techniques  used  to  collect 
data.    Data  analysis  was  described  in  terms  of  the  type  of  data  generated 
for  each  variable  (nominal,  ordinal,  interval  or  ratio),  the  statistical 
technique  employed  for  each  variable,  and  any  errors  discovered  in  either 
data  reduction  or  analysis.    Findings  (narrative  statements  of  numeric 
outcomes)  and  conclusions  (interpretations  or  explanations  of  the  findings) 
were  listed. 

Each  stu^y  was  subjected  to  tests  of  methodological  adequacy  be- 
cause of  the  necessity  to  have  assurance  of  the  credibility  of  the  findings 
of  each  study  included  in  the  synthesis.    It  was  reasoned  that  if  there  were 
methodological  flaws  in  the  study,  the  findings  and  conclusions  of  the  study 
might  not  be  valid  and  therefore  should  not  be  added  to  the  growing  know- 
ledge base  concerning  the  training  of  RDDE  personnel.    The  adequacy  of  studii 
was  judged  by  outside  experts  in  relation  to  each  of  the  following  eight 
aspects  of  a  study: 

1.  population  (clearly  defined?   appropriate  to  the  problem  being 
investigated?) 

2.  sample  (clearly  defined?   adequate  size  and  representativeness?) 

3.  variables  (clearly  delineated?  operationally  defined?  appro- 
priate to  the  problem  being  Invasti gated?) 

4.  Instrument  or  technique  used  for  data  collection  (objective? 
reliable?  valid?) 
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5.  response  rate  in  survey  studies  (sufficient  for  adequate 
analysis?   checks  for  non respondent  bias?) 

6.  data  analysis  (appropriate  statistical  techniques  employed? 
errors  which  significantly  alter  findings?) 

7.  findings  (accurate  translation  from  numeric  to  narrative  out- 
comes? unequivocal?) 

8.  conclusions  (warranted  on  the  basis  of  the  data  presented?) 

Finally,  judges  made  general  conments  about  each  study  and  a  recommendation 
as  to  whether  it  should  be  included  in  the  synthesis  of  studies  judged  to 

be  methodologically  sound.    (That  synthesis  is  contained  in  Technical 
Paper  No.  28  and  in  Chapter  VIII  of  this  report.) 

Each  study  was  judged  by  at  least  two  persons.   The  first  person  to 
judge  the  stu<fy  was  the  Task  Force  staff  member  who  had  written  the  descrip- 
tion of  the  study.   The  second  judge  was  an  outside  consultant  (i.e.,  some- 
one not  connected  directly  with  the  Task  Force)  knowledgeable  in  research 
design,  statistics,  measurement,  etc.    Where  there  were  differences  of  opinion 
between  the  first  and  second  judges,  a  third  judge  was  called  upon  to  resolve 
the  disagreement.    Finally,  a  member  of  the  AERA  Task  Force  served  as  a  con- 
sultant to  synthesize  the  judgments  made,  summarize  the  comments  made  on  the 
studies  by  the  judges,  and  aid  in  the  decision  of  whether  the  findings  and/or 

conclusions  of  each  study  should  be  included  in  the  synthrsis  of  results. 
There  were  no  manpower  studies  identified  which  failed  to  meet  the 

test  of  methodological  adequacy.   However,  there  were  several  studies  which 

were  excluded  from  the  review  process  which  led  to  the  synthesis  because 

the  studies  failed  to  meet  the  criteria  of  producing  generalizable  results 

and/or  having  enough  relevant  data  based  results  to  warrant  the  time  required 

to  describe  and  critique  the  total  study.   These  studies  included  the  twelve 

training  consortia  design  studies  submitted  to  the  U.  S.  Office  of  Education 
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in  December  1970.   Each  of  these  studies  contained  plans  for  establishing 

c>nd  implementing  programs  to  train  development  and  evaluation  personnel. 

Three  of  the  reports  pertain  directly  to  the  purpose  of  this  chapter  and 

include  results  of  surveys  on  the  perceived  need  for  RDDE  personnel  in 

selected  institutions,  agencies,  or  regions.    Therefore,  those  data  are 

reported  here.    It  should  be  noted  that  only  a  modified  version  of  the 

2 

description  and  judgment  phases  has  been  performed  on  these  studies. 

One  other  stuc|y  (the  Gideonse  report)  described  in  this  chapter  was 
included  in  Technical  Paper  No.  27  even  though  on  the  surface  it  did  not 
meet  the  criteria  for  inclusion  in  that  paper.    It  was  included  because 
it  was  felt  that  it  provided  the  most  comprehensive  overview  available  of 


The  procedure  for  describing  and  judging  the  training  program  reports 
included  the  following  steps.  A  member  of  the  Task  Force  staff  read  all  12  of 
the  reports  and  selected  those  three  which  contained  information  relevant  to 
the  topic  of  this  chapter.   The  relevant  sections  of  the  reports  were  described 
in  terms  of  both  the  methods  of  data  collection  and  the  results  of  the  data 
col lection. procedure.    Then  another  member  of  the  Task  Force  staff  read  those 
descriptions  so  that  a  consensual  judgment  of  the  validity  of  the  findings 
might  be  made.    The  external  validity  of  the  findings  was  deemed  questionable, 
but  for  the  purpose  of  this  chapter,  that  was  not  judged  to  be  a  sufficient 
reason  to  eliminate  the  findings.   In  terms  of  internal  validity,  it  was 
more  difficult  to  judge  the  adequacy  of  the  findings.    By  their  very  nature, 
the  three  reports  were  unlike  the  final  reports  of  most  rigorous  investiga- 
tions.   They  were  primarily  conceptual  design  activities  and  the  collection 
of  manpower  data  was  an  ancillary  activity  aimed  at  identifying  training 
needs.   The  results  of  these  manpower  needs  surveys,  reported  in  proposals 
submitted  for  further  funding,  were  evidently  intended  to  identify  general 
parameters  within  which  subsequent  training  would  take  place.  Consequently, 
many  of  the  details  and  the  level  of  precision  expected  in  most  final  research 
reports  are  lacking  in  the  reporting  of  some  of  the  results  in  the  three 
design  stu<fy  reports  included  here.   There  are,  therefore,  several  unanswered 
questions  which  limit  the  confidence  one  can  place  in  the  findings  of  these 
studies.   However,  it  was  felt  that  some  of  the  data  presented  in  the  three 
reports  were  worthy  of  inclusion  in  the  synthesis  of  results  section  of 
this  chapter  vf  the  reader  also  keeps  in  mind  the  limitations  in  these 
reports  discussed  above.   Specific  limitations  and  unanswered  questions  are 
pointed  out  as  they  arise  in  the  discussion  of  findings  which  follows. 
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American  educational  research,  development  and  related  activities.  Because 

one  section  of  the  report  was  concerned  with  manpower  supply  and  demand, 

the  Gideonse  paper  is  also  included  in  this  chapter. 

In  total,  then,  12  studies  dealing  with  manpower  were  reviewed  and  are 
3 

synthesized  herein.   They  include  the  following: 

Bargar,  R. ,  and  Hagan,  D.   A  survey  of  Ohio  public  school  administra- 
tors for  the  purpose  of  evaluating  the  undergraduate  research 
and  development  training  program.  1970. 

Clark,  D.  L.,  and  Hopkins,  J.  E.    A  report  on  educational  research, 
development,  and  diffusion  manpower,  1964-1974.  1969. 

Evans,  G.    The  need  for  research,  development,  dissemination,  and 
evaluation  personnel  in  education.    (Upper  Midwest  Regional 
Educational  Laboratory)  1970. 

Fleury,  B.    A  study  of  factors  relevant  to  the  development  of  applied 
educational  research  training  programs.  1968. 

Gideonse,  H.  D.    Educational  research  and  development  in  the  United 
States.  1969. 

Goodwin,  W.  L.,  and  Worthen,  B.  R.    An  interpretation  of  data  from 
the  AERA  employment  service,  1968-1970,  and  the  1969-70  telephone 
interviews:    Implications  for  supply  and  demand  and  emphases  in 
research  and  research -related  roles.  1970. 

Hood,  P, ,  et  al .    Design  of  a  functional  competence  training  program 
for  development,  dissemination  and  evaluation  personnel  at  pro- 
fessional and  paraprofessional  levels  in  education.    (Far  West 
Laboratory  for  Educational  Research  and  Development)  1970. 

Oldefendt,  S.  J.,  and  Worthen,  B.  R.    An  analysis  of  1969  AERA  em- 
ployment service  data:   Areas  of  specialization  reported  by 
applicants  and  areas  of  specialization  required  for  positions 
listed.  1970. 


^Five  more  studies  are  referenced  in  this  chapter.   They  are  Technical 
Paoers  13.  16,  24,  and  25,  and  the  Roadtn  Appendix  in  Clark  and  Hopkins  (1969). 
They  are  not  described  TTT  this  chapter  a.  the  above  12  reports  are  because 
onlv  one  finding  of  relevance  to  the  current  objective  was  found.    It  was  _ 
felt  that  adding  the  description  of  these  studies  would  lengthen  this  chap- 
ter urnrcessar??yr  all  of  the  other  reports  included  in  the  description  sec- 
tion of  this  report  include  many  more  than  one  finding  of  "^l^vance.  A 
complete  description  of  the  four  technical  papers  is  presented  in  Technical 
Paper  No.  27.  Vi^.^ 
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Manpower  needs  for  (1)  above  were  "compiled  by  the  project  staff 
with  the  aid  of  personnel  from  the  Minnesota  Department  of  Education.  Sev- 
eral local  school  district  officials  also  contributed  opinions  on  the  future 
needs  of  the  schools.   The  staff  and  consultants  first  estimated  the  roles 
and  skills  which  would  be  needed  by  local,  Intermediate  district,  and  state 
department  R»  D,  D,  and  E  personnel.    Then  an  estimate  was  made  of  the 
number  of  persons  likely  to  be  needed  In  each  of  these  categories  In  various 
school  district  size  categories"  (p.  3,  Design  Document  I). 

Needs  for  (2)  above  were  determined  by  sending  a  letter  to  31  RELs  and 
R  and  D  Centers.   In  this  letter,  a  request  was  made  for  a  copy  of  the  agency's 
organizational  chart,  job  descriptions  for  positions  shown  on  the  chart,  and 
a  summary  of  present  and  projected  research  personnel  needs  of  the  agency. 
"Less  than  one-half  of  the  agencies  responded.   Of  those  which  did  respond 
most  sent  an  organizational  chart  but  explained  that  no  position  descriptions 
were  available.   Only  a  few  sent  adequate  information,  giving  position  descrip- 
tions and  detailed  projections"  (pp.  24-25,  Dosign  Document  I).   As  a  result 
of  this  scanty  information,  no  projections  were  made  for  this  category. 

Needs  for  (3)  above  involved  the  following  processes  for  collecting 
estimates  of  the  numbers  of  research  personnel  likely  to  be  needed  in  institu- 
tions of  higher  learning: 

1)  A  list  of  all  the  colleges  and  universities  in  each^of  the  five 
states  in  the  region  was  compiled. 

2)  Project  staff  members  surveyed  "knowledgeable  college  administrative 
personnel"  (including  State  Departments  of  Education  personnel,  mem- 
bers of  the  State  College  Boards  and  Junior  College  Boards,  and 
various  college  and  university  research  directors)  regarding 
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Rogers,  W.  T.,  Worthen,  B.  R. ,  and  Sanders,  J.  R.   An  analysis  of 
1970  AERA  employment  service  data:    Competencies  reported  by 
applicants  and  competencies  required  for  positions  listed.  1970. 

Stufflebeam,  D.  L.   Proposal  to  design  new  patterns  for  training 
research,  development,  demonstration/dissemination,  and  evalua- 
tion personnel  in  education.    (The  Ohio  State  University)  1970. 

Worthen,  B.  R.    A  re-analysis  of  normative  data  from  the  National 

register  of  educational  researchers;    Career  patterns  of  research- 
ers  in  education  with  implications  for  recruitment.  1969. 

Worthen,  B.  R. ,  and  Sanders,  J.  F..    An  analysis  of  1968  AERA  enploy- 
ment  service  data:    Competencies  reported  by  applicants  and 
competencies  required  for  positions  listed.  1970. 

The  format  for  the  remainder  of  this  chapter  is  as  follows.  For  each 
stucly  included  in  the  synthesis,  a  very  brief  description  of  the  problem 
being  studied  and  the  way  in  which  it  was  studied  (i.e.,  data  generation  and 
analysis)  will  be  given.   Findings  and  conclusions  relevant  to  the  topic  of  this 
chapter     will  be  incorporated  in  the  synthesis  of  studies,  which  follows 
the  description. 

No  report  of  the  judgments  of  methodological  adequacy  made  about  the 
findings  and  conclusions  of  each  of  the  studies  will  be  given  here. 
The  reader  is  referred  to  Technical  Paper  No.  27  for  a  complete  listing  of 
those  judgments  for  all  of  the  studies  included  herein.    An  exception  is 
the  three  training  program  reports  which,  as  previously  reported,  were  judged 
by  two  Task  Force  staff  members  to  have  presented  findings  which,  with  quali- 
fications, were  worthy  of  inclusion  in  this  chapter.  It  should  also  be  pointed 
out  that  only  those  findings  and  conclusions  which  relate  to  the  topic  of 
manpower  supply  and  demand  in  educational  RDDE  are  included  in  this  synthesis. 
The  reader  is  again  referred  to  Technical  Paper  No.  27  for  a  complete  list- 
ing of  findings  and  conclusions,  and  to  Chapter  VIII  for  a  synthe- 
sis of  findings  and  conclusions  not^  related  to  manpower  supply  and  demand. 
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a)  number   and  type   of  employees  doing  institutional  R,  D,  D,  and 
E  activities  in  colleges  and  universities, 

b)  number   and  type   of  personnel  engaged  in  college  or  university 
R,  D,  D,  and  E  activities,  but  employed  at  the  state  level  (e.g., 
employees  of  State  College  Board), 

c)  projected  levels  of  employment  for  such  personnel  by  the  1972-73 
school  year. 

Needs  for  (4)  above  produced  no  data  pertinent  to  the  topic  under 
discussion  here. 

Synthesis  of  Findings 

This  section  of  the  chapter  includes  four  sub-sections:  (1)  findings 
relating  to  the  number  of  trained  persons  presently  in  RDDE  positions,  (2) 
findings  relating  to  potential  manpower  pools  for  RDDE  training,  (3)  findings 
relating  to  the  number  of  persons  being  prepared  to  fill  existing  and  emerg- 
ing RDDE  positions,  and  (4)  findings  relating  to  the  nunter  of  persons  who 
might  be  needed  by  the  RDDE  community  at  some  specified  future  date. 
Personnel  Prgsentlv  in  Educational  RDDE. 

The  only  estimate  of  the  size  of  the  nationwide  community  of  RDDE  per- 
sonnel was  made  by  Clark  and  Hopkins  (1969).    Based  on  data  from  Buswell  (1966), 
Bargar,et  a1 .  (1965),  Sieber  (1966)  and  others,  they  estimated  that  there  were 
4,125  full-time  equivalent  persons  engaged  as  professional  staff  menters  in 
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Description  of  Studies 

1.  Bargar,  R. ,  and  Hagan,       A  survey  of  Ohio  public  school  adminis- 
trators for  the  purpose  of  evaluating  the  undergraduate  research  and  develop- 
ment training  program.    In  Investigations  of  factors  influencing  the  training 
of  educational,  researchers.    Final  report  of  project  no.  3191,  U.  S.  Office 
of  Education,  contract  no.  OE -6-10-146,  1970. 

The  investigators  wished  to  determine  the  value  of  an  undergraduate 
training  program  in  educational  research  and  development  at  the  Ohio  State 
University,  particularly  with  respect  to  the  use  of  graduates  in  the  public 
school  systems  of  Ohio.   The  survey  was  designed  to  obtain  three  types  of 
information:    (1)  type   of  research  and  development  positions  presently 
existing  in  Ohio  public  schools;  (2)  an  assessment  of  the  number  of  positions 
which  might  be  open  to  graduates  of  the  training  program;  and  (3)  sugges- 
tions and  reactions  from  administrators  and  research  directors  concerning 
the  purpose  and  content  of  the  training  program. 

The  population  was  all  public  school  systems  in  Ohio.    A  questionnaire 
was  sent  to  a  sample  of  this  population;  the  sample  included  all  of  the 
larger  districts  in  the  state  plus  representative  medium  and  small  districts 
(N  =  100).   Sixty-seven  percent  of  the  diistricts  in  the  sample  returned 
completed  questionnaires.    Responses  were  analyzed  In  the  form  of  frequen- 
cies and  percentages. 

2.  Clark,  D.  L.,  and  Hopkins,  J.  E.    A  report  on  educational  re- 
search, development  and  diffusion  manpower^  1964-1974.   Blooming ton:  Indiana 
University  Research  Foundation,  1969. 

With  the  advent  of  the  Elementary  and  Secondary  Education  Act  of  1965, 
a  charge  was  made  to  create  a  community  of  researchers,  developers,  and  dis- 
seminators who  were  capable  of  bringing  research  directly  to  bear  on  public 
schools.    Existing  models  of  research  training  programs,  however,  were 
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research  and  research -related  activities  in  1964.^   Further,  they  estimated 
that  these  4,125  persons  were  divided  among  the  three  functions  of  research, 
development  and  diffusion  as  follows:    research  -  95.6%,  development  -  3.2%, 
'  and  diffusion  -  1 .2%. 

Clark  and  Hopkins  presented  three  projections  of  the  number  of  RDDE 
personnel  likely  to  be  required  in  1974.    The  most  optimistic  projection  was 
27,015,  the  most  likely  projection  was  19,436,  and  the  most  pessimistic  pro- 
jection was  12,827.   In  Chapter  VI  above,  Hopkins  revised  those  projec- 
tions on  the  basis  of  severely  reduced  funding  for  educational  RDDE  and  esti- 
mated that  approximately  8,669  RDDE  personnel  will  be  needed  in  1974.    If  one 


The  assumption  is  made  by  the  authors  that  Clark  and  Hopkins  used  the 
term  "professional  staff  member"  to  refer  to  a  person  at  the  doctoral  or 
master's  level.    This  assumption  is  based  on  the  fact  that  the  studies  cited, 
by  Clark  and  Hopkins  as  the  basis  for  their  baseline  projections  deal,  for 
the  most  part,  with  persons  completing  graduate  study.    Buswell  (1966)  con- 
ducted a  survey  of  educationists  who  had  received  their  doctorate  in  1954. 
In  the  National  Register  of  Educational  Researchers,  Bargar,et  al .  (1965)  re- 
port that  82.3%  of  the  persons  listed  hold  the  doctorate  and  16  percent  have 
a  master's  degree.   The  Sieber  stucjy  (1966)  is  somewhat  more  equivocal  in 
that  it  refers  only  to  "professional  personnel"  of  research  bureaus  and  insti- 
tutes.   Sieber  does  ss^y  that  "Most  of  the  professional  persons  associated 
with  research  units  are  non-staff  faculty  members  whose  work  is  facilitated 
by  the  unit"  (Sieber,  1966,  p.  95).  Based  on  the  support  given  by  the  Buswell 
and  Bargar  studies  ,  and  to  a  lesser  degree  by  the  Sieber  stuc|y,  the  assumption 
is  reiterated  that  Clark  and  Hopkins  use  the  term  "professional .  staff  member" 
to  refer  to  persons  at  the  doctoral  or  master's  level.    This  assumption  is 
important  because  many  of  the  findings  presented  later  in  this  chapter  report 
the  number  of  RDDE  personnel  at  the  different  academic  levels.    If  any  com- 
parisons are  to  be  made  between  the  Clark-Hopkins  findings  and  other  findings 
presented  herein,  then  it  is  necessary  that  the  above  assumption  be  accepted. 
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limited  In  number  and  scope.   The  heart  of  the  problem  appeared  to  be  the 
lack  of  Information  available  to  national  and  local  decision  makers  who 
would  be  called  upon  to  make  vital  decisions  In  regard  to  the  nature  of 
training  programs  to  be  supported  and  operated  under  Title  IV  of  ESEA. 
The  general  objective  of  this  study  was  to  develop  and  supply  data  useful 
to  policy  makers  and  planners  of  training  programs  for  research,  develop- 
ment, and  diffusion  personnel  In  education.   To  accomplish  this  objective, 
achievement  of  the  following  sub-goals  was  necessary:    (1)  project  the 
demand  for  existing  and  emerging  research,  development,  and  diffusion  roles 
In  education;  (2)  project  the  magnitude  of  the  demand  for  particular  types 
of  research,  development,  and  diffusion  personnel  In  education;  and  (3) 
project  the  Implications  of  these  data  for  the  recruitment,  selection,  and 
training  of  research,  development,  and  diffusion  personnel  In  education. 

To  accomplish  sub-goals  (1)  and  (2)  above,  the  following  procedures 
were  employed.    Data  (money  expended,  personnel  supported)  were  collected 
for  programs  and  projects  In  progress  In  FY  1966  In  the  USOE  support  pro- 
grams created  and  fostered  by  the  ESEA  of  1965.    The  same  data  we)-<e  also 
collected  forNSF's  Course  Content  Improvement  Program.   The  Instruments 
used  to  collect  much  of  these  data  were  the  proposals  for  programs  and 
projects  which  had  been  approved  for  funding  In  fiscal  year  1966.  These 
proposals  were  sampled  on  the  basis  of  this  objective:    "(1)  to  Identify 
all  of  the  diverse  populations  and  (2)  discover  all  of  the  various  types 
of  projects  which  existed  within  the  programs  selected  .  .  ."  (p.  131). 
Other  data  were  obtained  from  Interviews  with  administrators  of  the  several 
programs . 
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assumed  linear  growth  from  1964  to  1974,   the  most  likely  projection  of 
RODE  personnel  in  1971  based  on  the  original  Clark-Hopkins  projections 
would  be  14,843.    A  similar  projection  based  on  Hopkins 'revised  estimate 
would  be  about  half  that  number— 7,306  RDDE  personnel  in  1971. 

Several  smaller  studies  have  been  conducted  more  recently  to  deter- 
mine the  nuirber  of  professionals  and  paraprofessionals  in  RDC^  activities 
in  specific  agencies  or  geographical  regions.    It  is  reported  in  the  Far 
West  Laboratory's  training  program  proposal  (Hood,  et  al.,  1970)  that  985 
professionals  and  paraprofessionals  were  employed  in  educational  or  training 
RDDE  work  in  34  organizations  in  the  northern  California  area  (especially 
in  the  greater  San  Francisco  Bay  area).    Four  hundred  forty-six  of  these 
persons  were  at  the  doctoral  or  master's  level  (178  with  a  doctorate,  268 
with  a  master's  degree).    On  the  surface,  these  estimates  seem  reasonably  in 
1  ine  with  the  projections  of  Clark  and  Hopkins.    Bargar,  et  al .  (1965)  re- 
ported that  about  14  percent  of  his  national  sample  of  educational  researchers 
resided  in  the  Pacific  states  (California,  Oregon,  Washington,  Hawaii,  and 
Alaska).   This  provides  a  basis  for  comparison,  resulting  in  an  estimate  of 
between  959  and  1864  RDDE  personnel  (at  doctoral  or  master's  levels)  in  these 

^The  imposition  of  a  linear  growth  curve  on  these  data  may  do  an 
injustice  to  the  rationale  underlying  either  set  of  projections.  However, 
without  a  sufficiently  detailed  description  of,  how  the  projection  curve 
might  intercept  intermediate  years,  no  other  wiay  to  estimate  1971  personnel 
nunbers  is  readily  apparent.   If,  however,  growth  was  fastest  in  the  years 
ininediately  following  passage  of  the  ESEA  of  1965,  as  would  loqicallv  be 
expected  (and  as  implied  by  Hopkins  in  Chapter  VI),  then  the 
1971  projections  listed  above  are  likely  to  be  underestimated  to  some 
unknown  degree. 
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Data  reduction  and  analysis  procedures  are  too  complicated  to  be 
treated  In  anything  but  a  cursory  manner  here.   The  reader  Is 
referred  to  Technical  Paper  No.  27  or  to  the  original  document  Itself  for 
a  more  detailed  description  of  the  data  analysis.    In  essence,  what  Clark 
and  Hopkins  did  was  to  make  three  estimates  of  demand  for  trained  manpower 
In  educational  RDD  In  1974:   a  "least  optimistic"  projection,  a  "most 
optimistic"  projection,  and  a  "most  likely"  projection. 

The  subjects  of  Interest  In  the  attainment  of  sub-goal  (3)  were 
all  the  training  projects  funded  under  Title  IV  of  ESEA,  298  selected  pro- 
jects (of  1,145  total  projects)  funded  under  Title  III  of  ESEA,  and  all  of 
the  directors  of  Title  IV  research  training  programs  plus  340  (unspecified) 
professors  associated  with  research  training  programs. 

The  proposals  from  all  of  the  Title  IV  programs  were  reviewed  to 
determine  funding  for  the  programs,  number  of  trainees  In  the  programs, 
and  characteristics  of  the  training  programs.   A  questionnaire  was  sent 
to  the  Title  III  project  directors;  In  It  were  questions  regarding  charac- 
teristics of  Title  III  staff  members  and  directors  (e.g.,  age,  educational 
and  employment  history).    Usable  responses  were  received  from  137  Title  III 
centers  Involving  579  professional  staff  members.    A  letter  was  sent  to  the 
training  program  directors  and  faculty  In  which  Information  was  requested 
concerning  a  general  program  description.    Information  adequate  for  analysl 
was  received  from  47  Institutions  (there  Is  no  Information  as  to  how  many 
Institutions  received  the  letter).   Nonrespondent  bias  checks  were  not  dis- 
cussed.   Data  were  d(;3cr1bed  using  frequencies  and  means. 
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states  in  1970/   In  this  context,  446  doctoral  or  master's  RDDE  personnel 
in  Northern  California  seems  somewhat  high,  but  not  seriously  out  of  line 
since  that  region  is  one  of  the  most  populous  in  the  Pacific  states.  Such 
a  direct  comparison  of  data  ma^y  be  unwarranted,  however,  since  the  Far  West 
Laboratory  data  are  hard  to  interpret  for  at  least  three  reasons.  First, 
it  is  unclear  what  precise  kind  of  organization  is  included  in  the  group  of 
34  which  are  reported  as  employing  RDDE  personnel.   Also,  the  person  within 
each  organization  who  reported  the  number  of  RDDE  personnel  who  were  employed 
is  not  specified,  making  it  difficult  to  assess  the  probable  accuracy  of  the 
'  reports.    Finally,  no  operational  definition  of  "educational  or  training 
RDDE"  is  given.   It  is  very  likely  that  the  term  "educational  or  training 
RDDE,"  as  used  by  the  Far  West  Laboratory  group,  is  much  broader  than  (a)  the 
term  "educational  RDDE"  as  used  in  this  chapter  and  elsewhere  in  the  work  of 
the  AERA  Task  Force,  or  (b)  the  term  "educational  RDD"  as  used  by  Clark  and 
Hopkins.    If  this  is  true,  many  of  the  persons  reported  by  the  Far  West  Lab 
as  RDDE  personnel  ma^y  not  qualify  according  to  Task  Force  definitions  (see 
Technical  Paper  No.  1)  or  definitions  of  Clark  and  Hopkins  as  being  a  meiriber 
of  the  educational  RDDE  community. 

The  Ohio  State  training  program  proposal  (1970)  gives  two  figures  for 
the  number  of  persons  currently  employed  in  RDDE  activities.    Based  on 

^These  figures  are  computed  by  the  following  methods:    (a)  apply  the 
linear  growth  curve  to  both  earlier  sets  of  projections  to  arrive  at  estimates 
in  1970  of  13,312  from  the  oriainal  Clark-Hopkins- projections  and  6,851  from 
Hopkins 'revised  projection;  (b)  assume     RDDE  personnel  in  1970  would  be 
distributed  regionally  about  the  same  as  were  Bargar's  group  of  educational 
researchers  in  1964;  and  (c)  take  14  percent  of  each  of  the  estimates  in  (a) 
above  to  result  in  estimates  of  1864  from  the  Clark-Hopkins  projections  and 
959  from  the  Hopkins  revised  projection. 
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Although  the  findings  and  conclusions  of  this  study  will  be  des- 
cribed In  another  section  of  this  chapter,  there  Is  one  more  detail  of  the 
Clark-Hopkins  study  which  should  be  mentioned  now.   That  detail  Is  the  one 
general  assumption  underlying  all  of  the  projections  made  In  the  stuc(y: 

.  .  the  educational  R,  D,  and  D  community  will  not  experience  events  which 
will  alter  Its  situation  to  any  greater  extent  than  did  the  ESEA  of  1965. 
In  other  words,  the  general  assumption  Is  made  that  major  political,  social, 
and  economic  trends  of  the  recent  past  will,  with  one  caveat,  continue.  The 
caveat  Is  that  the  limitations  Imposed  upon  program  growth  and  expansion  dur- 
ing fiscal  years  FY  '67-'69  are  viewed  as  unusually  stringent  and  will  not  be 
continued  during  the  period  . FY  '71-'74"  (pp.  120-121).    Recent  history  has 
shown  that  this  assumption  has  not  held.   The  financial  limitations  imposed 
on  program  growth  beginning  In  FY  1967  have  remained  to  the  present,  and  In 
some  cases  have  become  even  more  stringent.    An  updated  version  of  the 
Clark-Hopkins  projections,  based  on  recent  funding  histories.  Is  presented 
In  Chapter  VI  of  this  report.    The  reader  Is  referred  to  that  chapter  for 
a  revised  estimate  of  the  number  of  educational  RDDE  personnel  who  might 
be  supported  In  1974. 

3.  Fleury,  B.  A  study  of  factors  relevant  to  the  development  of 
applied  educational  research  training  programs.  (Doctoral  dissertation. 
University  of  Massachusetts),  1968. 

This  study  was  composed  of  three  parts.    Part  I  was  an  examination  of 
the  Applied  Educational  Research  Training  Program  at  the  University  of  Massa- 
chusetts.   Part  II  was  a  study  of  the  research  training  programs  funded  under 
ESEA  Title  IV.    Part  III  was  an  Investigation  of  potential  employers  of  applied 
educational  researchers.       Because  Part  I  is  an  evaluation  of  one  specific 
program,  it  is  excluded  from  further  discussion. 
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interviews  conducted  with  consortium  members  and  others,  it  was  reported 
that  there  were  390  RDDE  personnel  (133  with  a  doctorate,  16  with  an  MA) 
employed  in  the  consortium  member  agencies  (The  Ohio  State  University  Devel- 
opment Faculty;  The  Ohio  State  University  Evaluation  Center;  The  Center  for 
Vocational  and  Technical  Education,  The  Ohio  State  University;  Cincinnati 
City  School  System;  Saginaw,  Mich.  City  School  System;  Illinois  Institute 
of  Technology  Research  Institute;  Montana  State  Department  of  Education; 
Rhode  Island  State  Department  of  Education;  Dallas  City  Schools).  From  the 
survey  of  regional  laboratories  it  was  learned  that  there  were  65  educa- 
tional development  specialists  (16  with  a  PhD,   23  with  an  MA)  employed 
in  the  labs.    Here  again  it  is  necessary  to  point  out  some  problems  asso- 
ciated with  these  data.   Although  it  was  stated  that  interviews  were 
conducted  with  representatives  of  all  consortium  member  organizations  and 
various  other  individuals  who  were  knowledgeable  in  the  field  of  educational 
RDDE,  the  list  of  interviewees  provided  in  the  proposal  does  not  appear  to 
include  a  person  from  each  consortium  agency.    Yet  when  the  figures  given 
above  are  presented,  a  footnote  explains  that  the  numbers  reflect  a  compila- 
tion of  all  agencies  in  the  consortium.    It  is  unknown  how  information  was 
obtained  for  the  agencies  not  represented  by  an  interviewee.   Before  one  can 
properly  interpret  the  regional  lab  data,  it  is  necessary  to  know  what  is 
meant  by  an  educational  development  specialist.   According  to  the  question- 
naire sent  to  the  directors  of  the  RELs,  an  educational  development  specialist 
is  a  person  who  possesses  a  constellation  of  skills  and  training  such  as  the 
fol 1 owi  ng : 
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The  population  for  Part  II  of  the  stucly  was  the  85  ESEA  Title  IV 
research  training  programs.*  Data  were  collected  from  grant  request  pro- 
posals, program  brochures,  and  college  and  university  catalogs  of  the  spon- 


soring institutions.    Variables  studied  were  the  roles  of  educational  re- 
search and/or  the  researcher,  the  academic  levels  of  training  for  educa- 
tional research,  and  specific  information  about  the  training  programs  (e.g., 

» 

entrance  requirements,  course  requirements,  apprenticeship  experiences).  The 
variables  studied  were  given  the  name  "key  aspects  of  educational  research 
training." 

The  population  for  Part  III  was  the  50  state  departments  of  education, 
the  superintendents  of  Massachusetts  school  districts,  and  15  (unspecified) 
major  independent  research  institutes.   These  groups  were  surveyed  for  the 
following  information:   their  views  on  the  appropriateness  of  various  levels 
of  academic  training  for  educational  researchers,  their  previous  pattern 
of  employing  educational  researchers,  their  views  on  training  curricula  and 
field  experiences,  and  their  expectations  for  the  employment  of  educational 
researchers.    Ninety-four  percent  of  the  state  departments,  73  percent  of 
the  institutes,  and  77  percent  of  the  district  superintendents  responded  to 
the  questionnaire. 

A  weighting  system  was  devised  so  that  the  information  gathered  could 
be  translated  into  recomnendations  for  the  University  of  Massachusetts  train- 
ing program.    If  a  "key  aspect"  received  13  points  according  to  the  weighting 

Jneury  gives  the  number  of  Title  IV  Graduate  Research  Training  Programs 
as  85;  however,  there  are  89  such  programs  (see  Hopkins,  et  al .  1970).    It  is 
unclear  why  Fleury  uses  data  on  only  85.   
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Skills 

1.  Provide  information  for  program  planning  and  design 

2.  Engineer  prototypic  packages  (innovative  solutions  and 
programs)  for  educational  problems 

3.  Create  widespr*ead  awareness  of  the  tested  solutions  and 
programs 

4.  Monitor  and  adjust  the  so"!  tions  and  programs  after  wide- 
spread installation 

5.  Conduct  basic  scientific  inquiry 

6.  Test  and  evaluate  innovative  solutions  and  programs 

7.  Train  target  audiences  in  the  use  of  solutions  and  programs 

8.  Assess  educationally  related  needs  and  problems 

9.  Invent  solutions  to  operating  problems 

10.    Demonstrate  the  effectiveness  of  solutions  and  programs 


Training 


1. 

Statistics 

IK 

Advertising 

2. 

Learning  theory 

12. 

Political  science 

3. 

Social  psychology 

'J 

13. 

Philosophy  of  science 

4. 

Audio- visual 

14. 

Data  processing 

5. 

Marketing 

15. 

Theories  of  teaching 

6. 

Sociology 

16. 

Curriculum  development 

7. 

Research  design 

17. 

Communication  theory 

8. 

Testing  and  measurement 

18. 

Decision  theory 

9. 

Systems  theory 

19. 

Information  theory 

10. 

Educational  administration 

20. 

Journalism 
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system,  the  practice  was  highly  reconmiended  for  adoption  by  the  training 
program;  if  it  received  10-12  points,  it  was  recommended  for  adoption;  if 
it  received  6-9  points,  it  was  considered  experimental  and  recommended  for 
immediate  investigation;  if  it  received  0-5  points,  it  was  recommended  for 
investigation  at  some  later  point. 

A  practice  ("key  aspect")  could  amass  points  in  the  following  manner: 
if  the  practice  was  favored  (i.e.,  if  it  was  the  modal  response  to  an  item) 
by  the  three  employer's  groups,  it  received  one  point  per  employer  group; 
three  points  were  assigned  to  the  practice  if  it  was  the  modal  response  of 
the  Title  IV  trainers;  seven  points  were  assigned  if  all  the  available  re- 
search supported  the  practice. 

One  problem  encountered  in  the  author's  treatment  of  the  data  concerns 
the  variables  for  which  no  data  were  gathered  frc?n  the  employers.    In  this 
case,  the  author  weighted  the  training  program  response  double  what  it  would 
normally  be  according  to  the  weighting  scheme.   There  seems  to  be  no  rationale 
for  this  practice.    Another  problem  is  that  with  variables  on  which  the  em- 
ployers and  trainers  differ,  the  author  appears  to  arrive  at  an  arbitrary 
resolution  of  the  difference.   Because  of  these  problems,  no  attempt  will  be 
made  in  the  synthesis  section  of  this  chapter  to  present  any  of  the  point  totals 
for  any  key  aspect.    Responses  to  each  question  by  training  program  directors 
and  employers  will  simply  be  reported. 

4.    Gideonse,  H.  D.    Educational  research  and  development  in  the 
United  States.   Washington,  D.  C:    U.S.  Office  of  Education,  1969. 

This  report  serves  as  a  review  of  the  development,  current  status,  and 

possible  future  of  educational  research  and  development  in  the  United  States. 

The  Gideonse  study  is  organized  around  the  following  major  areas  of  concern: 
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Operationally,  this  seems  to  be  an  extremely  broad  definition  of  a  special- 
ist in  educational  development.    At  least  it  is  at  variance  with  most 
current  definitions  of  development  proposed  either  by  theoreticians  (e.g., 
see  Technical  Papers  No.  1  and  4  and  Clark  and  Hopkins,  1969)  or  practi- 
tioners in  curriculum  development  projects.   Many  of  the  skills  listed  above 
would  more  likely  be  possessed  by  the  researcher  (e.g.,  #5),  the  diffuser 
(e.g.,  #3,  #7),  or  the  evaluator  (e.g.,  #1)  than  by  the  developer.  Therefore, 
it  is  necessary  for  the  reader  to  avoid  the  mistake  of  using  the 
terms  "developer"  and  "educational  development  specialist"  interchangeably. 
The  latter  term  is  a  broader  one  and  may  perhaps  better  be  used  as  a  synonym 
for  the  total  range  of  RDDE,  as  used  in  this  report. 

Both  the  Ohio  State  and  Far  West  Laboratory  surveys  indicate  that 
there  are  a  substantial  number  of  sub-doctoral  personnel  employed  in  RDDE 
activities.    It  is  unclear,  however,  whether  these  sub-doctoral  personnel  (a) 
have  actually  had  any  training  in  educational  RDDE,  (b)  are  employed  primarily 
because  not  enough  educational  RDDE  doctorates  are  available  (although  other 
data  will  be  presented  later  which  cast   doubt  on  this  possibility) ,  or  (c) 
even  need  specialized  training  to  perform  their  duties.    If  the  sub-doctoral 
personnel  Identified  in  these  surveys  do  indeed  heed  specialized  training, 
one  must  wonder  how  they  are  obtaining  that  training.    Roaden  (1969)  lists  47 
universities  and  other  institutions  which  offer  a  total  of  54  research  train- 
ing programs.    Only  five  of  those  54  programs  prepare  personnel  at  the  master's 
level  and  six  at  the  undergraduate  level.   Worthen,  et  al.  (in  press)  reported 
that  of  the  89  Title  IV  graduate  research  training  programs  in  1969,  only  nine 
prepared  personnel  at  the  master's  level,    rf  the  above  hypothesis  is  correct-- 
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1.  The  definition  of  educational  research  and  related  functions, 
and  possible  models  for  the  relationships  among  those  functions. 

2.  The  organization  of  educational  research  in  the  U.S.,  according 
to  both  its  sponsors  and  its  performers. 

3.  The  management  of  educational  research  and  development. 

4.  Financial  and  manpower  resources. 

5.  The  substance  of  educational  R  and  D,  analyzed  along  six  dimen- 
sions (e.g.,  the  type  of  research  function  supported,  topical  areas  of 
stucly,  special  characteristics  of  target  groups),  and  its  impact  on  educa- 
tional policy  practice. 

6.  Educational  policy  implications  from  R  and  D  outcomes. 

The  section  of  the  Gideonse  report  which  is  concerned  with  manpower 
resources  is  the  only  section  which  is  relevant  to  the  topic  of  this  chapter. 
Thus,  the  other  sections  are  eliminated  from  the  discussion  here. 

Gideonse  used  the  Clark  and  Hopkins  (1969)  study  to  establish  a  begin- 
ning estimate  of  the  manpower  supply  for  educational  research.    In  order  to 
supplement  the  Clark-Hopkins  estimates,  Gideonse  uses  two  additional 
sources  for  estimates  of  manpower  in  related  areas.   Those  two  other  sources 
are  the  National  Register  of  Scientific  and  Technical  Personnel  (1966)  and 
USOE  reports  on  earned  degrees  and  degree  candidates  in  disciplines  related 
to  educational  ROD.    Finally,  Gideonse  turns  to  specific  educational  research 
manpower  development  activities  of  USOE  to  gather  figures  on  the  nmber  of 
persons  being  trained  annually  in  educational  research  and  research -related 
activities. 
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that  sub-doctoral  personnel  do  need  specialized  training  In  RDDE— then  It 
Is  clear  that  more  training  programs  designed  to  prepare  RDDE  personnel  at 
the  sub-doctoral  level  are  needed.  The  three  consortium  training  projects 
recently  funded  by  USOE  (Ohio  State,  Far  West  Lab,  Learning  R  and  D  Center) 
should  help  to  fill  this  need. 

Manpower  Pools  Available  for  Recruitment  Into  Educational  RDDE 

It  has  been  argued  that  there  alreac(y  exist  several  manpower  pools 

which  might  be  tapped  to  meet  the  demand  for  educational  RDDE  pen^onnel . 

Several  studies  have  examined  the  size  of  those  manpower  pools  and  the  degree 

to  which  certain  of  them  might  be  more  susceptible  to  recruitment  than  others. 
Gideonse  (1969)  reports  the  number  of  doctorates  conferred  In  1966-67 

In  the  fields  of  psychology,  anthropology,  sociology,  economics  and  llnguls- 

D 

tics,  plus  ten  percent  of  the  education  doctorates,    as  2,663.   The  number 
of  doctorates  in  the  same  categories  expected   In  1967-68  was  3,358.    He  also 
reports  that  there  are  22,475  doctoral  and  11,914  master's  level  persons  In 
psychology,  economics,  sociology,  anthropology  and  linguistics  among  the 
persons  listed  In  the  National  Register  of  Scientific  and  Technical  Personnel. 
Gideonse  draws  no  conclusions  regarding  the  potential  contribution  to  educa- 
tional RDDE  of  any  of  these  persons,  but  the  presentation  of  these  figures 
does  imply  the  existence  of  a  fairly  large  manpower  pool  In  disciplines  relat- 
ed to  education  which  might  be  available  for  research  and  research-related 

8 

Ten  percent  of  the  total  number  of  doctorates  In  education  were 
chosen  on  the  grounds  that  this  Is  a  fair  approximation  of  "research"  degrees 
In  education. 
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5.   Worthen,  B.  R.   A  re-analysis  of  normative  data  from  the  National 
register  of  educational  researchers;   Career  patterns  of  researchers  in  educa- 
tion with  implications  for  recruitment.    In  D.  L.  Clark  and  J.  E.  Hopkins, 
A  report  on  educational  research,  deyelogment.  and  diffusion  manpower.  1964- 
1974.   Bloomington:    Indiana  University  Research  Foundation,  1969. 

There  were  two  unrelated  problems  investigated  in  this  stutly.  The 

first  problem  concerned  the  career  patterns  of  educational  researchers:  what 

positions  had  these  people  held  before  they  entered  the  research  conmunity? 

The  second  problem  investigated  was  a  specific  aspect  of  the  research  training 

program— the  apprenticeship  experience.    The  following  questions  were  posed 

in  the  stuc(y: 

1.  Are  there  manpower  pools  susceptible  to  recruitment  [to  the  field 
of  educational  research]  which  are  not  being  tapped? 

2.  Why  is  there  a  ".  .  .  wide  discrepancy  between  the  stated  accept- 
ance of  apprenticeship  as  a  crucial  training  component  and  the  failure  to 
provide  training. opportunities  to  a  great  many  prospective  researchers?" 

3.  "Are  the  relatively  few  research  assistantships  now  available  being 
utilized  to  provide  the  type  of  experience  which  is  most  valuable  to  pros- 
pective researchers?" 

Because  the  second  problem  (and  questions  2  and  3  above)  is  unrelated 
to  the  topic  of  manpower  projections,  it  will  be  eliminated  from  further 
discussion  here. 

The  population  was  composed  of  the  5,121  persons  who  identified  them- 
selves with  the  educational^search  conmunity  by  returning  written  responses 
to  a  questionnaire  submitted  to  them  by  Bargar,  and  who  in  so  doing  were 
•  included  in  the  National  Register  of  Educational  Researchers.   Worthen  collected 
no  new  data  of  his  own;  the  questionnaires  collected  by  Bargar  in  his  stucty 
(Bargar,  Guba,  and  Okorodudu,  1965)  provided  the  data  for  Worthen's  study. 
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activities  in  education.   Whether  or  not  recruitment  of  such  personnel 
into  educational  RDDE  can  be  conducted  effectively  is  still  a  moot  point 
that  will  be  resolved  only  by  further  research  or  actual  attempts  to  make 
such  recruitment  work. 

Worthen  (1969)  looked  at  manpower  pools  already  in  the  educational 
community  to  see  which  of  ^these  might  be  most  susceptible  to  recruitment  into 
educational  RDDE.   He  reasoned  that  if  educational  researchers  tend  to  have 
certain  professional  background  experiences  more  often  than  others  (i.e.,  if 
they  were  more  often  public  school  teachers  than  guidance  counselors,  for  ex- 
ample, before  they  moved  into  research  positions),  then  recruitment  efforts 
should  be  specifically  aimed  at  those  occupational  groups  which  most  often 
appear  in  early  stages  of  researchers'  career  backgrounds.^ 

Public  school  teachers  appear  to  be  the  most  promising  group  from 
which  potential  educational  researchers  might  be  recruited. They  appeared 


It  should  be  noted  that  sometimes  a  group  was  considered  in  this  study 
as  both  an  immediate  and  a  long-term  recruitment  group  for  a  certain  position. 
This  occurred  when  the  data  showed  that  personnel  in  a  certain  research  posi- 
tion often  moved  from  one  job  such  as  a  public  school  teacher  to  another  job 
as  an  academic  instructor  to  another  job  as  a  research  staff  member,  but  might 
just  as  often  have  moved  directly  from  the  position  of  public  school  teacher 
to  research  staff  member.    In  such  a  situation,  public  school  teachers  would 
be  considered  both  an  immediate  and  a  long-term  target  group  for  the  position 
of  research  staff  member. 

^^It  should  be  kept  in  mind  that  the  focus  here  is  on  recruitment  into 
RDDE  training  programs,  not  recruitment  into  RDDE  itself.    Whereas  some  of  the 
persons  in  Gideonse's  lists  probably  possess  sufficient  skills  to  move  directly 
into  RDDE  work  in  education,  this  is  unlikely  to  be  true  for  many  public  school 
teachers  or  persons  in  several  of  the  categories  discussed  below. 

It  should  also  be  noted  that,  given  the  current  oversupply  of  teachers 
in  many  areas,  persons  graduatint)  with  teaching  certificates  may  be  more  suscep 
tibia  than  ever  to  recruitment  into  RDDE  training  programs. 
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For  the  first  objective  (to  identify  manpower  pools  susceptible 
to  recruitment  efforts  by  looking  at  the  career  patterns  of  persons  now 
in  educational  research),  the  investigator  excluded  from  his  sample  all 
persons  in  the  population  who  devoted  less  than  20  percent  of  their  total 
professional  time  to  research.   Those  questionnaires  which  remained  were 
then  sorted  into  16  categories  on  the  basis  of  the  major  position  of  the 
respondent.    Any  respondent  who  could  not  readily  be  assigned  to  one,  and 
only  one,  category  was  excluded  from  the  analysis.   Data  from  1  ,755  persons 
remained  available  for  stu^y. 

The  variables  studied  were  census-type  data  (sex,  present  position, 
percentage  of  time  spent  in  research  activities,  major  area  and  date  of 
highest  degree)  and  a  sequential  listing  of  previous  professional  positions 
(including  position  title,  name  of  institution   aid  Inclusive  dates  of  em- 
ployment).   Chi -square  tests  of  significance  were  applied  to  the  data.  The 
professional  position  data  were  tabulated  by  <:ategories. 

6.   Worthen,  B.  R. ,  and  Sanders,  J.  R.    An  analysis  of  1968  AERA 
employment  service  data:    Competencies  reported  by  applicants  and  competen- 
cies required  for  positions  listed.    Technical  Paper  No.  6.  Boulder, 
Colorado:   AERA  Task  Force  on  Research  Training,  1970. 

Oldefendt,  S.  J.,  and  Worthen,  B.  R.    An  analysis  of  1969  AERA 
employment  service  data:   Areas  of  specialization  reported  by  applicants  and 
areas  of  specialization  required  for  positions  listed.   Technical  Paper  No.  7. 
Boulder,  Colorado:    AERA  Task  Force  on  Research  Training,  1970. 

Rogers,  W.  T.,  Worthen,  B.  R.,  and  Sanders,  J.  R.    An  analysis  of 
1970  AEP't  employment  service  data:    Competencies  reported  by  applicants  and 
competencies  required  for  positions  listed.    Technical  Paper  No.  8.  Boulder, 
Colorado:   AERA  Task  Force  on  Research  Trjining,  1970. 

Goodwin,  W.  L.,  and  Worthen,  B.  R.    An  interpretation  of  data  from 
the  AERA  employment  service,  1968-70,  and  the  1969-70  telephone  interviews: 
Implications  for  supply  and  demand  and  emphases  in  research  and  research- 
related  roles.    Technical  Paper  No.  10.   Boulder,  Colorado:    AERA  Task  Force 
on  Research  Training,  1970. 
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to  be  an  immediate  target  group  for  recruitment  into  positions  as  research 
staff  members  in  public  schools;  they  appeared  to  be  long-range  target 
groups  to  fill  the  following  positions: 

a.  research  directors  in 

1 )  public  schools 

2)  state  education  agencies 

3)  professional  associations 

4)  private  research  agencies 

5)  college  and  university  institutional  research  units,  research 
programs,  and  educational  research  bureaus 

b.  research  staff  members  in  public  schools 

c.  stimulators  and  coordinators  of  RDDE  activities 

d.  Individual  research  personnel  in  schools  of  education. 

Another  promising,  and  perhaps  more  immediate,  target  group  for  re- 
cruitment comprises  academic  instructors.    They  may  be  considered  an  immediate 
target  group  for  the  following  positions: 

a.  research  directors  in  college  and  university  institutional  research 
units,  research  programs ,' and  educational  research  bureaus 

b.  research  staff  members  in 

1)  private  research  agencies  ^ 

2)  college  and  university  research  programs 

c.  stimulators  and  coordinators  of  ROUE  cjctivities 

d.  individual  research  personnel  in 

1)  schools  of  education 

2)  other  discipl  iiu^;  and  atddtMinc  departments. 

Academic  instructors  also  appear  to  be  a  long-range  target  recruitment 
group  for  the  positions  of: 
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The  purpose  of  the  first  three  studies  listed  above  was  to  provide 
needed  information  about  the  supply  of  and  demand  for  trained  personnel  in 
educational  RDDE.   To  accomplish  this  objective,  analyses  were  made  of  ex- 
isting data  from  the  placement  service  of  the  1968,  1969  and  1970  conventions 
of  the  American  Educational  Research  Association.   Technical  Paper  No.  10 
was  an  attempt  to  integrate  data  from  these  three  studies  into  a  meaningful 
analysis  of  manpower  trends.    Because  of  the  interrel atedness  of  these  four 
papers,  this  abstract  deals  with  them  collectively. 

Application  forms  (standard  U.  S.  Department  of  Labor  convention  employ- 
ment service  forms)    and  employer  vacancy  forms  from  all  prospective  employees 
and  employers  registered  with  the  placement  service  at  the  AERA  annual  meet- 
ings, and  task  Force  forms  on  which  applicants  and  employers  listed  competen- 
cies possessed  and  needed  were  the  sources  of  data  for  the  studies.  For 
the  applicant,  the  variables  studied  were  type  of  position  sought,  his  area(s) 
of  competence,  and  the  type  of  organization  in  which  he  sought  employment; 
for  the  employer,  the  variables  were  type  of  position  available  and  competen- 
cies required  by  a  person  to  fill  that  position.  The  operational  definition 
of  "type  of  position"  was  RDDE  position,  RDDE  facilitative  position,  and  non- 
RDDE  position.   Tabulation  of  vacancies  and  applicants  by  "competency"  and 
"type"  was  presented,  as  well  as  category  percentages. 

7.   Training  Program  Proposals. 

The  three  ptx)posals  which  contained  data  deemed  relevant  to  this  chap- 
ter were  those  submitted  by  The  Ohio  State  University,  the  Far  West  Labora- 
tory for  Educational  Research  and  Development,  and  the  Upper  Midwest  Regional 
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a.  research  directors  in  college  and  university  institutional 
research  units  and  research  programs 

b.  research  staff  members  in 

1)  private  research  agencies 

2)  college  and  university  research  programs 

c.  individual  research  personnel  in  other  disciplines  and  academic 
departments . 

Public  school  administrators  form  an  immediate  target  recruitment 
group  for  the  position  of  research  director  in  (1)  public  schools,  (2) 
state  education  agencies,  (3)  professional  associations,  (4)  private 
research  agencies,  and      an  intermediate  target  group  for  the  same  posi- 
tions in  college  and  university  bureaus  of  educational  research. 

Research  assistants  serve  as  an  intermediate  target  recruitment  group 
for  the  position  of  research  director  in  private  research  agencies.  They 
'  may  be  considered  both  a  prime  immediate  and  long  term  recruitment  group  for 
the  position  of  research  staff  member  in  (1)  private  research  agencies  and 
(2)  college  and  university  research  programs. 

As  was  previously  noted,  persons  in  all  of  these  groups  may  only.be 
considered  potential  recruits  to  the  educational  RDDE  community.  Without 
further  research  or  actual  attempts  to  recruit  these  personnel,  no  definitive 
statements  may  be  made. 

One  might  question  the  need  to  recruit  persons  to  the  field  of  educa- 
tional research  when  apparently  all  present  positions  in  educational  RDDE 
can  easily  be  filled  from  the  present  pool  of  RDDE  trainees.    Data  reported 
in  Technical  Papers  No.  6,  7,  and  8  of  the  Task  Force  series  (and  discussed  in  a 
later  section  of  this  chapter)  indicate  this  to  be  the  case— there  may  in- 
deed be  an  oversupply  of  trained  RDDE  personnel.    If  this  is  true,  one  may 
even  argue  it  is  unethical  to  recruit  more  persons  into  a  field  where  there 
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Educational  Laboratory.   The  parts  of  the  proposals  concerned  with  man- 
power supply  and  demand  will  be  described  separately  below. 

A.   The  Ohio  State  University.   This  consortium  group  conducted  a 
context  evaluation  to  determine  the  current  state  of  educational  RDDE  activi- 
ties and  needs.    Included  in  the  context  evaluation  were  two  methods  of  data 
collection  which  provided  information  on  the  supply  of  and  demand  for  train- 
ed personnel  in  educational  RDDE.    Those  two  methods  were: 

1)  ".  .  .an  interview  guide  structured  to  gain  a  wide  range  of 
knowledge  relating  to  RDD  and  E  activities  now  being  conducted 
by  various  agencies  and  activities  that  they  perceive  their  agen- 
cies engaged  in  for  the  future.   All  consortium  members  were 
interviewed  and  in  addition  various  individuals  who  had  exper- 
tise in  the  areas  of  research,  development,  diffusion,  evaluation, 
environmental  analysis,  and  management  were  contacted.  .  .  .  The 
interview  guide  .  .  .  provides  input  regarding  needs  for  the 
training  program,  opportunities  for  training,  and  problems  that 
might  be  encountered"  (pp.  9-11,  Vol.  III). 

2)  "A  survey  of  the  fifteen  regional  educational  laboratories  was 
conducted  in  an  effort  to  provide  two  types  of  data.    Two  survey 
instruments  were  developed:-   Professional  Employee  Profile  and 
Director's  Questionnaire.    The  Employee  Profile  sought  data  relat- 
ing to  their  work  activities  and  training.    The  D.icpctor's  Question- 
naire sought  data  relating  to  the  activities,  training  areas, 
employment  trends,  and  needs  for  educational  specialists  within  the 
educational  laboratory  network M p.  140,  Vol.  HI).    A  total  of  13 
Director's  Questionnaires  and  280  Liiiploy^  Profiles  (representing 
12  RELs)  were  returned.!  fl A       ,  \ 
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are  not  enough  jobs  to  begin  with.   However,  such  an  argument  ignores  tv^o 
facts.    First,  the  fact  that  there  may  be  a  current  oversupply  of  RDDE 
trainees  in  education  does  not  necessarily  mean  that  there  is  an  oversupply 
of  talented  RDDE  trainees  in  education.    It  is  probable  that  continued 
vigorous  recruitment  efforts  aimed  at  attracting  able  young  trainees  into 
the  field  would  have  large  dividends  in  the  conduct  of  quality  RDDE  work. 
Second,  the  fact  that  there  is  a  current  oversupply  of  trainees  should  not 
create  pessimism  about  the  future  of  educational  RDDE  as  a  career  possibility. 
The  proposed  National  Institute  for  Education  and  other  developing  RDDE  pro- 
grams may  well  require  additional  RDDE  personnel  for  the  future.    These  trends 
are  unclear,  but  it  would  seem  unwise  to  assume  the  market  for  RDDE  trainees 
would  remain  saturated  for  a  prolonged  period. 

Present  Trainees  in  Educational  RDDE 

Even  though  there  appear  to  be  extant  manpower  pools  which  might  be 
susceptible  to  recruitment  efforts  by  members  of  the  educational  RDDE  commun^ 
ity,  the  first  source  of  manpower  to  fill  existing  and  emerging  RDDF  positions 
is  obviously  the  group  of  persons  who  are  currently  being  trained  in  RDDE. 
Several  studies  have  been  conducted  which  present  information  on  the  number 
of  trainees  in  current  training  programs. 

Gideonse  (1969)  reported  the  number  of  persons  involved  in  all  of  the 
Title  IV  training  programs  (undergraduate,  graduate,  postdoctoral,  and  insti- 
tutes) in  the  years  from  1966  to  1969.    In  1966,  2,542  persons  received  some 
sort  of  training  in  educational  RDDE  through  Title  IV  programs  of  all  types; 
in  1967,  the  number  was  1,934;  in  1968,  it  was  2,291;  and  in  1969,  it  was 
2,579.    It  should  be  noted  that  1967  was  the  last  year  that  the  undergraduate 
research  training  program  was  in  existence.   The  fact  that  the  total  number 
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B.  Far  West  Laboratory.   "In  order  to  establish  a  more  detailed 
estimate  of  regional  demand  and  to  identify  prospective  employers  an  exten- 
sive effort  was  undertaken  to  establish  who  were  the  employers  of  educational 
and  training  RDD  and  E  personnel  in  northern  California,  and  especially  in 
the  greater  San  Francisco  Bay  area.    Names  of  prospective  employers  were  com- 
piled from  personal  knowledge  of  consortium  members,  fron;  professional  and 
organizational  directors,  from  the  Foundations  Directory  (1970),  and  from 
li.S.O.E.  Current  Projects  Information  (July,  1970). 

"Through  October  31  [1970],  115  organizations  and  firms  were  contacted, 
with  34  reporting  that  they  employed  personnel  in  educational  or  training  R, 
0.  D  or  E"  (p.  09,  Vol.  2). 

C.  Upper  Midwest  REL.    "The  data  presented  in  this  report  does  not 
represent  a  survey  of  what  agency  directors  say  they  need,  but  judgments  of 
informed  experts  about  personnel  likely  to  be  needed  in  various  activities 
for  each  category  of  agency.    Care  was  taken  to  find  individuals  who  were 
informed  and  in  the  mainstream  of  current  agency  activity.  .  .  .  The  need 
study  deals  with  four  separate  areas  or  groups  of  agencies  where  R,  D,  D,  and 
E  personnel  are  likely  to  be  needed:  (1)  local,  intermediate,  and  state  educa- 
tion agencies,  (2)  Regional  Laboratories  and  Research  arid  Development  Centers, 
(3)  colleges  and  universities,  and  (4)  educational  or  training  divisions  of 
industrial  firms. 

"The  prime  concern  was  to  find  data  relating  to  needs  in  the  five  state 
area  (Minnesota,  Iowa,  North  Dakota,  South  Dakota,  Wisconsin).   However,  where 
other  data  became  readily  available  it  was  also  included  -.  .      (pp.  1-2,  De- 
sign Document  I). 
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of  persons  receiving  training  remained  relatively  stable  after  that  is 
accounted  for  by  the  increased  number  of  participants  in  institutes. 
Numbers  of  persons  in  the  graduate  and  postdoctoral  training  programs 
remained  constant  during  1968  and  1969. 

In  reviewing  trainee  numbers,  it  should  be  kept  in  mind  that  there 
is  not  a  direct  ratio  between  the  numbers  presented  by  Gideonse  and  the 
number  of  "new"  members  of  the  educational  RDDE  community.  Participants 
in  the  postdoctoral  and  institute  programs  are  likely  to  be  mostly  persons 
who  are  already  practitioners  in  the  field  who  return  to  gain  more  special- 
ized knowledge.  Only  those  trainees  graduating  from  the  Graduate  Research 
Training  Programs  each  year  are  new  additions  to  the  field.    Gideonse  lists 
the  number  of  Title  IV-supported  trainees  as  732  in  1966,  974  in  1967,  809 
in  1968,  and  809  in  1969.    Hopkins,  et  al .  (1970)  listed  927  trainees  in  the 
Graduate  Research  Training  Programs  during  the  1969-70  academic  year.^^ 

Since  the  doctoral  program  is  typically  a  three  year  program,  it 
seems  reasonable  to  assume  that  about  300  persons  graduate  from  the  programs 


and  enter  the  RDDE  field  each  year.    A  check  of  the  progress  reports  for  1969- 
70  (which  contain  lists  of  graduates  during  that  year)  shows  this  assumption 
to  be  correct  at  least  for  1969-70. 


Not  all  of  the  927  persons  listed  as  trainees  are  supported  by  Title 
IV  funds.    What  Hopkins,  et  al.  (1970)  refer  to  as  the  "ripple  effect"  of 
Title  IV  training  programs  accounted  for  118  nonstipend  trainees  being  pre- 
pared during  that  year  along  with  the  809  Title  IV-supported  trainees  in  the 
89  research  training  programs.    It  is  the  opinion  of  Hopkins,  et  al .  that 
"It  is  doubtful  if  a  large  proportion  of  the  nonstipend  group  would  be  re- 
ceiving systematic  training  in  educational  research  in  the  absence  of  the 
Title  IV  training  programs"  (Hopkins,  et  al.,  1970,  p.  15).    If  this  opinion 
is  correct,  then  the  phasing  out  of  the  Title  IV  Graduate  Research  Training 
Programs  (which  will  be  discussed  later  in  this  chapter)  causes  even  more  of 
a  loss  of  potential  RDDE  personnel  than  would  be  thought  by  looking  only  at 
the  number  of  Title  IV-supported  trainees. 
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There  is  corresponding  information  in  the  Clark  and  Hopkins  (1969) 
report.   They  say  that  "The  real  quantitative  effect  [of  the  Title  IV 
programs]  rests  in  the  (1)  Graduate  Research  Training  and  (2)  the  Institute 
and  Special  Training  Project  areas"  (p.  304).   Their  estimate  of  the  num- 
ber of  RDDE  personnel  who  will  be  graduated  each  year  from  the  graduate 
training  programs  is  between  300  and  400.    Assuming  continuance  of  the 
funding  level  of  1968,  they  projected  the  total  number  of  newly  trained 
personnel  between  1966  and  1974  to  be  2650.   Of  course,  it  is  now  apparent 
that  the  funding  level  of  the  Title  IV  training  programs  has  not  been  con- 
tinued at  expected  levels.   Several  Graduate  Research  Training  Programs 
were  eliminated  in  1970  and  all  are  being  phased  out  at  present.  The 
total  amount  of  money  to  be  spent  on  training  activities  was  cut  in  FY  '71 
from  $6.25  million  to  $2  million,  with  the  further  proviso   that  the  $2 
million  was  to  be  spent  on  newly  announced  projects  rather  than  on  existing 
programs  (Educational  Researcher.  September  1970,  p.  10). 

The  three  newly  developed  training  consortia  which  have  recently  been 
funded  by  USOE  will  be  training  a  total  of  approximately  225  persons  at  the 
graduate  level  (doctoral  and  master's)  over  the  next  three  years. These 
persons  will  be  receiving  training  in  evaluation  (the  thirty  doctoral  and 
ninety  master's  level  students  at  Ohio  State),  development  (the  36  students 

12 

In  the  proposals  submitted  by  The  Ohio  State  University  and  the 
Far  West  Laboratory,  projected  numbers  of  trainees  were  given  for  the  dura- 
tion of  the  program  (three  years  for  the  Far  West.  Lab  program,  three  and 
one  half  yt.ars  for  the  Ohio  State  program).   Learning  Research  and  Develop- 
ment Center  (LRDC) ,  however,  listed  only  the  expected  number  of  trainees  for 
the  first  year  of  the  program.    In  order  to  compare  the  training  consortia 
programs  with  the  Title  IV  Graduate  Research  Training  Programs,  it  was 
necessary  to  simply  triple  the  number  of  trainees  attributed  to  the  LRDC 
program.    This  may  be  an  injustice  to  LRDC,  for  it  may  be  their  intention 
to  expand  their  program  and  train  more  persons  in  the  second  and  third  years. 
HoWever,  in  the  absence  of  further  information,  it  was  impossible  to  make 
overall  numerical  projections  without  making  this  assumption. 
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at  the  graduate  level  at  LRDC) ,  and  diffusion  (the  fifty  to  eighty  master's 
level  students  at  the  Far  West  Lab,  who  will  also  be  receiving  training 
in  evaluation  and  development).    The  training  consortia  programs  will  also 
be  providing  training  in  development,  diffusion,  and  evaluation  to  approxi- 
mately 475  paraprofessionals.   Together,  then,  the  training  consortia  will 
be  producing  approximately  700  newly  trained  personnel  over  the  next  three 
or  three  and  one  half  years.    Remembering  that  the  Graduate  Research  Train- 
ing Programs  have  been  producing  about  300  newly  trained  personnel  per  year 
(for  a  total  of  about  900  over  the  next  three  years),  it  can  be  seen  that 
the  number  of  educational  RDDE  persons  entering  the  field  from  Federally 
supported  training  programs  will  be  reduced  by  about  200  overall  over  the 
next  three  years.    However,  if  one  excludes  paraprofessionals,  the  reduction 
at  the  graduate  level  is  more  severe  —  a  reduction  of  675  in  all,  or  225 
per  year.   This  means  that  the  shift  in  support  from  the  original  Title  IV 
programs  to  the  training  consortia  will  result  in  only  25  percent  as  many 
graduate  trainees  per  year  as  has  previously  been  the  case.    Although  addi- 
tional consortia  may  be  funded,  it  appears  at  the  moment  that  the  emphasis 
is  clearly  moving  away  from  graduate  training  in  research  to  training  para- 
professionals and  some  graduates  in  educational  DD  and  E. 

Apparently  the  short  term  training  programs  supported  under  Title  IV 
will  be  seriously  reduced  if  not  completely  eliminated,  resulting  in  a  loss 
of  retraining  or  short  term  training  for  approximately  1500  persons  per  year. 
The  three  training  consortia  will  be  providing  short  term  trainingjn  D,  D, 
and  E  to  approximately  1100  persons  over  the  next  three  or  three  and  one- 
half  years.   This  is  far  short  of  the  approximately  4500  served  by  Title  IV 
in-service  programs  during  the  same  time  period.    Since  the  necessity  of 
keep,1ng  up  to  date  on  current  developments  in  the  field  is  quite  pronounced 
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in  educational  RDDE,  this  could  jeopardize  the  quality  of  work  in  these 
areas  by  providing  insufficient  retraining  or  short  term  training  to  up- 
grade skills  and  prevent  obsolesence.    Again,  since  Clark  and  Hopkins 

(1969,  p.  329)  report  that  short  term  training  opportunities  were  primarily 
1 3 

in  research,     this  represents  a  dramatic  shift  in  training  away  from  re- 
search and  toward  development,  diffusion,  and  evaluation. 

Task  Force  Technical  Papers  No.  6,  7.  8  and  10  provide  some  infor- 
mation on  the  number  of  persons  looking  for  jobs  in  educational  RDDE.  At 
the  1968  annual  meeting  of  the  American  Educational  Research  Association, 
there  were  811  persons  registered  as  applicants  with  the  placement  service; 
in  1969,  the  number  of  applicants  dropped  to  569;^^  in  1970,  it  rose  again 
to  727.   The  applicants  were  differentiated  on  the  basis  of  whether  they 
were  seeking  a  RDDE  position,  a  RDDE-facilitati ve  position,  or  a  position 
not  related  to  RDDE.    This  differentiation  resulted  in  the  finding  that  the 
number  of  persons  seeking  RDDE  or  RDDE-facili tative  positions  fell  from  655 
in  1968  to  516  in  1970,  while  the  number  seeking  nonresearch  positions  rose 
markedly  from  121  in  1968  to  210  in  1970.    In  1968,  there  were  2.03  applicants 

1 3 

Of  35  institutes  funded  during  fiscal  year  1966,  25  provided  train- 
ing in  research,  8  in  development,  and  2  in  diffusion. 

14 

It  is  not  possible  to  be  certain,  but  one  factor  which  may  have 
influenced  this  reduction  was  the  site  of  the  1969  annual  meeting.   The  con- 
vention was  held  in  Los  Angeles,  whereas  AERA  draws  larger  proportions  of  its 
membership  from  the  upper  Midwest  than  from  other  sections  of  the  country. 
The  expense  of  traveling  to  the  West  Coast  may  have  been  a  deciding  factor  — 
especially  for  students  —  in  whether  or  not  they  attended  the  convention.  It 
is  reasonable  to  assume  that  the  fewer  students  there  are  at  the  convention, 
the  fewer  persons  there  will  be  registered  with  the  placement  service. 
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for  every  RDDE  or  RDDE-facilitati ve  position  vacancy  listed  with  the 
placement  service;  in  1970,  the  ratio  had  dropped  to  1.79  applicants  for 
every  vacancy  in  those  two  categories.    Given  the  recent  scarcity  of 
positions,  this  trend  seems  to  run  counter  to  the  general  increase  in 
number  of  applicants  per  position  opening.    Of  course,  one  cannot  assume, 
given  a  tight  job  nrarket,  that  all  persons  applying  for  RDDE  or  RDDE- 
facilitative  positions  are  sufficiently  trained  for  these  positions  (or 
have  actually  received  RDDE  training  per  se) .   However,  since  some  300 
trainees  graduated  from  the  Title  IV  Graduate  Research  Training  Programs 
alone  each  year  during  this  period,  it  seems  reasonable  to  assume  that 
most  of  the  applicants  did  graduate  from  programs  which  provided  them  with 
relevant  RDDE  training. 

Fleury's  study  (1968)  provides  some  information  on  the  type  of  train- 
ing being  provided  to  persons  in  the  Graduate  Research  Training  Programs. 
He  found  that  94.1%  of  the  programs  were  training  personnel  for  research 
positions,  23.0%  were  training  personnel  for  development  positions,  and 
5.8%  were  training  personnel  for  diffusion  positions.   This  would  partially 
corroborate  a  conclusion  reached  by  Clark  and  Hopkins  (1969)  that  even 
though  there  appears  to  be  extensive  fluctuation  in  the  nature  of  the  de- 
mand for  RDD  personnel,  ".  .  .  the  response  of  the  field  has  been  to  repli- 
cate in  its  training  programs  the  proportions  of  personnel  found  in  the 
1964  R,  D,  and  D  community"  (p.  422).    It  will  be  remembered  that  those 
proportions  were  95.6%,  3.2%,  and  1.2%  for  research,  development,  and 
diffusion,  respectively.   The  noticeable  deviation  from  the  estimates  is 
in  the  area  of  development,  where  the  emphasis  on  training  is  seven  times 
that  reported  by  Clark  and  Hopkins. 
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In  summary,  it  appears  that  up  to  the  present  time  there  have  been 
approximately  300  graduates  from  the  Graduate  Research  Training  Programs 
each  year.   The  relative  emphasis  on  training  researchers,  developers, 
and  diffusers  has  been  in  the  ratio  of  16:4:1  (Fleury,  1968)  Since 
over  500  persons  registered  with  the  AERA  placement  service  in  1970  and 
indicated  that  they  were  seeking  RDDE  or  RDDE-facilitative  positions,  there 
apparently  are  over  200  persons  who  view  themselves  as  trained  in  educa- 
tional R,  D,  D,  or  E  who  have  no  connection  with  the  Title  IV  Graduate  Re- 
search Training  Programs.    Of  course,  some  of  those  200  are  persons  presently 
holding  RDDE  or  RDDE-facilitative  positions  who  are  seeking  advancement  in 
the  field.   But  there  are  likely  to  be  others  among  those  200  who  are  enter- 
Ing  the  field  of  educational  RDDE  for  the  first  time  and  who,  presumably, 
have  some  degree  of  competence  in  R,  D,  D,  or  E.   Some  data  on  non-Title  IV 
research  training  programs  are  presented  in  Chapter  V  of  this    report.  Of 
83  institutions  which  responded  to  a  questionnaire  and  indicated  that  they 
had  research  training  programs,  53  were  not  connected  with  the  Title  IV 
Graduate  Research  Training  Program.    The  53  non-Title  IV  programs  are  pre- 
sently training  443  persons  at  the  subdoctoral  level  (full  and  part  time 
students)  and  495  persons  at  the  doctoral  level  (full  and  part  time).  Thus 
it  would  appear  that  the  phasing  out  of  the  Title  IV  programs  will  not  dry 
up  the  supply  of  newly  trained  RDDE  personnel  entering  the  field.    If  indeed 
there  are^a  large  number  of  persons  being  trained  in  non-Title  IV  programs 
and  if  the  training  consortia  programs  do  produce  the  estimated  700  persons 
over  the  next  three  years  as  anticipated,  it  appears  that  the  supply  of 
trained  RDDE  personnel  will  still  be  at  least  adequate  when  compared  to  the 

15 

Unfortunately,  evaluators  were  hot  differentiated  by  Fleury.  Most 
probably  they  were  grouped  in  eijther  the  research  or  development  categories. 
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number  of  positions  presently  available  in  the  current  depressed  job 
market  for  educational  RDDE,   Of  course,  if  the  National  Institute  for 
Education  comes  into  existence,  the  job  market  may  "boom"  again  (as  it 
did  immediately  after  passage  of  the  ESEA  in  1965). 

The  foregoing  discussion  has  been  concerned  simply  with  numbers 
of  trainees.   The  data  presented  could  be  interpreted  to  argue  that  there 
are  presently  too  many  trained  RDDE  personnel  for  too  few  jobs.  However, 
this  completely  ignores  the  question  of  quality   of  the  trainees.    As  was 
argued  in  the  preceding  section  on  manpower  pools  available  for  recruit- 
ment, attempts  must  be  continued  to  attract  the  most  able  persons  into 
training  programs  and  provide  them  with  quality  training  in  all  four  in- 
quiry and  inquiry-related  functions  in  education.   The  impact  educational 
RDDE  will  have  on  improving  education  will  be  dependent  on  the  success  of 
the  training  efforts. 

Future  RDDE  Personnel  Needs 

A  projection  is,  by  definition,  the  carrying  forward  of  a  trend  into 
the  future.      To  make  projections  of  manpower  demand,  one  may  look  at  finan- 
cial trends  and  transform  money  figures  into  personnel  figures,  or  one  may 
ignore  the  financial  aspect  and  look  simply  at  the  perceived  need  for  personnel 
at  some  future  point  in  time.    Studies  of  manpower  supply  and  demand  in  edu- 
cational RDDE  have  been  conducted  in  both  ways;  this  section  includes  those 
studies  relating  to  the  demand  for  trained  RDDE  personnel. 

Clark  and  Hopkins  (1969)  present  the  only  projection  of  demand  based 
on  financial  data.  Their  intent  was  to  determine  the  amount  of  money  which 
would  be  available  to  support  RDDE  personnel  in  1974.    Then,  based  on  the 
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assumption  that  the  amount  of  money  available  for  RDD  would  detemine 
the  number  of  persons  who  would  be  employed  in  RDD,  a  projection  was  made 
of  the  number  of  RDD  professional  staff  members  who  would  be  needed  by  FY 
74.^^   As  mentioned  earlier  in  this  paper  in  the  description  of  the  Clark- 
Hopkins  study,  three  projections  were  made:    a  least  optimistic  projection, 
a  most  optimistic  projection,  and  a  most  likely  projection.    The  smallest 
number  of  personnel  which  would  be  needed  by  FY  '74  according  to  Clark  and 
Hopkins  was  12,827;  they  estimated  27,015  as  the  maximum  which  might  be 
supported  in  1974.  The  most  probable  number  of  RDD  personnel  who  would  be 
needed  and  could  be  supported  was  estimated  at  19,436.   This  last  figure 
represents  almost  a  five-fold  increase  over  the  number  of  meiribers  in  the 
educational  RDD  community  which  Clark  and  Hopkins  said  existed  in  1964. 
However,  other  data  cause  at  least  mild  doubt  about  the  accuracy  of  the 
Clark-Hopkins  projections.    One  can  note  that  the  total  number  of  vacancies 
listed  with  the  AERA  placement  service  (Goodwin  and  Worthen,  1970)  fell 
from  769  in  1968  to  459  in  1969  and  to  412  in  1970. 

It  should  be  emphasized  that  the  Clark -Hoplhfls  projections  were 
based  on  the  assumption  that  the  ".  .  .  major  political,  social,  and  economic 
trends  of  the  recent  past  will  .  .  .  continue"  (Clark  and  Hopkins,  1969, 
p.  120).    Viewed  from  the  vantage  point  of  late  1971,  it  can  be  seen  that 
that  assumption  has  not  held.    Hopkins,  in  the  preceding  chapter  of  this 
report,  makes  a  new  projection  of  8,669  persons  needed  by  FY  '74,  based  on 
current  funding  patterns.   This  is  considerably  below  even  the  least 
optimistic  of  the  earlier  Clark-Hopkins  projections. 

^Ss  in  the  Fleury  §tudy  (1968),  evaluators  were  not  differentiated 
in  the  CI  ark -Hopkins  report.    They  most  likely  were  grouped  with  either  the 
researchers  or  the  developers. 
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Clark  and  Hopkins  not  only  projected  a  dramatic  Increase  between 
1964  and  1974  In  the  numbers  of  RDD  personnel,  they  also  projected  a  marked 
shift  In  the  functions  of  those  personnel.    Whereas  1n  1964  the  proportions 
of  research,  development  and  diffusion  personnel  were  95.6%,  3.2%,  and  1.2%, 
respectively,  by  FY  '74  those  proportions  were  projected  to  be  33  percent 
for  research,  50  percent  for  development,  and  17  percent  for  diffusion. 

There  Is  some  supporting  evidence  In  the  AERA  placement  service  data 
that  the  need  for  developers  Is  In  fact  Increasing.    At  the  AERA  placement 
service,  "Although  applicants  for  positions  In  development  continue  to  out- 
number vacancies  .  .  .  there  are  some  bits  of  Information  which  suggest  that 
development  Is  emerging  as  a  more  Important  and  larger  occupational  area  than 
was  previously  true.  ...  It  was  noted  that  there  was  a  modest  increase  in 
demand  In  1970  for  persons  having  competencies  In  educational  development, 
curriculum  development/analysis,  and  Instructional  media/techniques" 

(Goodwin  and  Uorthen,  1970,  p.  50). 

There  appears  to  be  no  support  at  this  time  (at  least  In  the  AERA 

placement  service  data)  for  the  Clark  and  Hopkins  projection  that  the  need 
for  diff users  will  Increase  about  14  times  from  1964  to  1974.  Indeed, 
".  .  .  diffusion  as  an  area  of  competence  essentially  was  Ignored  by 
employers  and  applicants  over  the  two  years  examined''  (Goodwin  and  Worthen, 
1970,  p.  20). 

It  was  found  In  the  analysis  of  AERA  placement  service  data  (Goodwin 
and  Worthen,  1970)  that  there  seems  to  be  an  oversupply  of  applicants  with 
research  skills  (e.g.,  educational  research,  research  design,  testing/ 
applied  measurement,  statistics).   This  Is  not  surprising  In  view  of  Fleury's 
(1968)  finding  that  94.1%  of  the  graduate  research  training  programs  are 
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training  personnel  to  fill  research  positions.   However,  the  overs upply 

was  by  a  factor  of  about  four  and  one-half  and  would  indicate  that  training 

in  these  areas  might  profitably  be  reduced  by  80  percent,  not  eliminated, 

as  seems  the  case  in  the  present  shift  of  Federal  priorities. 

There  is  some  support  in  the  placement  service  data  (Goodwin  and 

Worthen,  1970)  which  may  indicate  that  Clark  and  Hopkins  were  correct  when 
they  projected  that  the  demand  for  research  personnel  would  decrease.  "In 
1968  .  .  .  24.8  percent  of  the  vacancies  required  competency  in  research 
methods/types,  while  this  figure  dropped  to  12.3  percent  of  the  vacancies 
in  1970.  .  .  .  The  overall  decrease  in  measurement  from  13.6  percent  to  7.9 
percent  was  particularly  evident  in  the  research-related  and  research- 
facilitative  categories.   A  dramatic  reduction  also  occurred  in  the  percentage 
of  vacancies  requiring  competence  in  statistical  analysis  (11.8  percent  in 
1968  to  2.1  percent  in  1970)"    (Goodwin  and  Worthen,  1970,  pp.  19-20). 

In  terms  of  over-all  RDDE  supply-demand  trends,  some  information  is 
available  from  non-financially  based  supply -demand  data,  such  as  that  avail- 
able from  the  AERA  annual  employment  service.    For  example,  although  the 
nunfeer  of  applicants  per  vacancy  in  research  and  research-facilitati ve 
categories  decreased  from  2.03  applicants  per  vacancy  in  1968  to  1.79  appli- 
cants per  vacancy  in  1970,  "...  the  AERA  employment  service  data  seem  to 
contain  early  indicators  of  what  could  develop  into  a  'depressed  job  market' 
situation.    For  example  .  .  .  it  can  be  noted  that  applicants  outnumber 
vacancies  in  all  three  categories  (directly  related  to,  facilitative  of,  and 
not  related  to  RDDE).   Additionally,  fewer  transactions  are  occurring  on  the 
job  market;  the  absolute  numbers  of  both  vacancies  and  applicants  are  down. 
In  comparing  the  1968  and  1970  data,  this  is  particularly  true  for  vacancies 
(769  to  412)  and  less  pronounced  fdr  applicants  (811  to  727)"  (Goodwin 
and  Worthen,  1970,  p.  48).  r  t^A 
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Looking  at  demand  only,  four  studies  have  been  conducted  In  which 
data  are  presented  on  the  perceived  need  for  trained  RDDE  personnel  In 
specific  geographical  regions.   Bargar  and  Hagan  (1970),  In  their  survey 
of  public  school  administrators  In  Ohio,  found  that  those  administrators 
responding  to  thelv^  questionnaire  on  personnel  needs  were  quite  positive 
In  their  Interest  In  and  support  for  the  undergraduate  research  training 
program  at  the  Ohio  State  University.    In  answer  to  the  question  "Do 
you  feel  that  graduates  [of  the  undergraduate  research  training  program] 
would  be  able  to  qualify  for  positions  In  the  research,  development,  or 
evaluation  activities  of  your  department  or  district?"   affirmative  res- 
ponses were  given  by  76  percent  of  the  respondents  In  large  districts,  80 
percent  In  medium  districts,  and  80  percent  In  small  districts.  However, 
when  asked  "Do  you  have  positions  now  open  for  which  graduates  of  this 
program  might  qualify?"    73  percent  of  the  large  districts,  75  percent  of 
the  medium  districts,  and  90  percent  of  the  small  districts  responding 
answered  "no."   In  answer  to  the  question  "Do  you  anticipate  any  openings 
for  1969-70  for  which  graduates  of  this  program  might  qualify?"   56  percent 
of  the  large  districts,  100  percent  of  the  medium  districts,  and  83  percent 
of  the  small  districts  responding  answered  "no."    Thus,  It  seems  that 
although  the  administrators  would  like  to  have  trained  RDDE  personnel  em- 
ployed In  their  districts,  there  appears  to  be  little  real  demand  for  (or 
little  money  to  support)  such  personnel  to  assume  actual  positions.  The 
stringent  financial  situation  which  caused  Hopkins  to  revise  his 
earlier  projection  downward  from  19,435  trained  RDDE  personnel  in  FY  '74  to 
8,669  very  probably  is  a  significant  factor  in  the  lack  of  positions  avail- 
able in  the  Ohio  public  schools  as  well. 
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The  Ohio  State  proposal  (1970)  included  the  results  of  a  survey  of 
manpower  needs  in  Regional  Educational  Laboratories.   One  of  the  questions 
in  that  survey  asked  REL  directors  for  their  estimate  of  the  percentage 
increase  by  1980  in  the  emfsloyment  of  educational  development  special- 
ists in  R  and  D  centers,  REL>,  public  schools,  universities  and  colleges, 
private  research  institutes,  medical  institutions,  military  institutions, 
business  and  industrial  concerns,  state  education  agencies,  and  the  USOE. 
As  a  basis  for  the  estimate,  each  REL  director  was  asked  to  assume  that 
each  of  those  categories  of  agencies  presently  employs  100  educational 
development  specialists.    The  mean  percentages  reported  in  answer  to  that 
question  ranged  from  a  78  percent  increase  in  the  number  of  educational 
development  specialists  likely  to  be  employed  by  medical  institutions  to 
a  600  percent  increase  in  the  employment  of  such  personnel  in  the  RELs. 
The  percentage  increase  in  six  of  the  ten  institutional  settings  (R  and 
D  Centers,  public  schools,  universities  and  colleges,  private  research 
institutes,  state  education  agencies,  and  USOE)  ranged  from  120  percent 
to  310  percent.    Three  institutional  settings  ranged  from  78  percent  to 
83  percent. 

The  REL  directors  were  also  asked  to  estimate  their  needs  for  edu- 
cational development  specialists  in  their  own  agency  in  1975  and  1980.  By 
1975,  the  directors  estimated  there  will  be  a  131  percent  increase  in  the 
nunber  of  doctoral  level  development  specialists  (from  the  16  presently 
employed  to  37).    By  1980,  they  predicted  a  further  increase  of  68  percent 
to  62  doctoral  level  development  specialists.   At  the  master's  level,  the 
directors  predicted  by  1975  a  100  percent  increase  (from  the  23  presently 
employed  to  46)  and  a  further  43  percent  increase  (to  a  total  of  66)  by 
1980. 
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Interviews  conducted  by  Ohio  State  personnel  also  yielded  data  on 
the  number  of  full  time  equivalent  RDDE  personnel  perceived  as  needed  by 
1975  in  the  agencies  included  in  the  consortium  group.  Representatives 
of  agencies  said  that  in  1975  there  would  be  a  need  for  268  doctoral  level 
RDDE  persons  (an  increase  of  135  over  the  number  presently  employed)  and 
392.5  (FTE)  master's  level  persons  (an  increase  of  226.5  over  the  current 
number  employed)^  It  should  be  noted  again  that  it  is  unclear 
how  these  nunfcers  were  reached  for  the  entire  consortium  group  when  it 
did  not  appear  from  the  proposal  that  every  consortium  agency  was  repre- 
sented by  an  interviewee. 

In  the  Far  West   Lab's  stucly,  the  34  organizations  in  the  northern 
California  area  presently  employing  RDDE  personnel  reported  that  they  would 
likely  need  an  additional  129  persons  at  the  doctoral  and  master's  level  by 
1971-72.   This  finding  led  the  proposal  authors  to  state  that  "These  results 
are  preliminary,  but  they  obviously  suggest  that  the  market  for  trained 
personnel  is  substantially  larger  than  might  be  estimated  if  only  a  narrow 
concept  of  'educational'  R  and  D  is  taken"  (p.  DIO,  Vol.  2).    The  under- 
estimates made  by  others  are  especially  true  in  the  business  and  industrial 
sector,  in  the  opinion  of  the  Far  West  Lab.   The  caution  issued  earlier  re- 
garding the  Far  West  Lab's  use  of  the  term  "educational  RDDE"  should  be 
repeated  here. 

Estimates  are  made  in  the  training  proposal  submitted  by  the  Upper 
Midwest  REL  (UMREL)  for  the  need  for  RDDE  personnel  at  the  professional  and 
supervisory  levels  (Ph.D. and  M.A.  levels)  in  school  districts,  state  depart- 
ments of  education,  and  state  intermediate  units  in  the  5-state  region  of 
Minnesota,  North  Dakota,  South  Dakota,  Wisconsin,  and  Iowa.   Those  estimates 
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for  the  number  of  RDDE  personnel  needed  by  1972-73  are:    2,526  in  school 
districts,  144  in  state  departments  of  education,  and  228  in  state  inter- 
mediate units.    No  figures  are  given  for  current  employment  of  RDDE  per- 
sonnel \n  these  agencies,  so  no  statement  may  be  made  concerning  the  amount 
of  growth  perceived  as  likely  to  occur.    Little  information  is  provided  in  the 
proposal  on  how  the  estimates  were  made,  other  than  to  say  that  the  project 
staff,  with  the  aid  of  the  Minnesota  Department  of  Education,  compiled  the 
information  and  made  the  estimates.   Due  to  this  lack  of  information,  the 
estimates  lose  some  of  their  credibility. 

The  UMREL  report  also  Included  some  generalizations  about  the  number 
of  RDDE  personnel  likely  to  be  needed  at  colleges  and  universities  in  the 
five-state  region,  but  they  appear  to  be  rather  weak.    The  only  way  one  may 
estimate  from  these  UMREL  data  the  future  need  for  RDDE  personnel  in  univer- 
sities and  colleges  is  by  multiplying  the  number  of  colleges  in  the  region 
in  each  category  they  list  (e.g.,  junior  college,  university)  by  the  esti- 
mated staffing  level  they  list  for  that  type  of  college  by  1973.    The  authors 
of  the  proposal  fail  to  give  an  estimate  of  the  staffing  level  in  the  uni- 
versity category,  however,  saying  that  the  estimated  staffing  level  of  RDDE 
personnel  depends  on  the  size  of  the  university.   While  this  is  no  doubt 
true,  without  further  information  it  is  impossible  to  estimate  the  need  for 
trained  personnel  in  this  category  --  likely  the  category  employing  the 
greatest  number  of  RDDE  personnel  in  education.    Without  the  inclusion  of 
the  number  of  RDDE  personnel  needed  by  the  universities,  this  REL's  esti- 
mate pf  personnel  needed  in  institutions  of  higher  education  (177  in  1973) 
has  little  value  for  the  purposes  of  this  chapter. 
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In  attempting  to  summarize  the  information  presented  in  this 
section  of  the  paper,  the  one  central  thread  which  runs  through  the  results 
of  all  the  studies  is  the  equivocal  nature  of  the  data  reported.  Collect- 
ively, these  data  do  not  provide  much  guidance  for  the  trainer  of  RDDE 
personnel.    In  the  Clark  and  Hopkins  (1969)  study,  the  manpower  projec- 
tions (of  a  need  for  between  12,000  and  27,000  RDDE  personnel  in  1974)  were 
based  on  a  range  of  anticipated  funding  levels  which  failed  to  materialize, 
thus  invalidating  all  three  sets  of  projections.   Hopkins  uj^dated  the 
original  projections  on  the  basis  of  new  estimates  of  future  funding  levels, 
resulting  in  an  estimate  of  8,669  RDDE  personnel  needed  in  1974  (see  Chapter 
VI).    Although  this  estimate  is  believable,  it  suffers  from  two  serious  inade- 
quacies:   (a)  it  is  not  coupled  with  an^  indication  of  how  many  educational 
RDDE  positions  or  persons  presently  exist,  making  it  impossible  to  determine 
whether  we  currently  have  more  or  fewer  than  8,669  persons  employed  in 
the  field,  and  (b)  Hopkins  was  unable  to  break  out  functional  emphases 
in  his  revised  projections,  thus  providing  no  Information  on  the  relative 
emphasis  that  should  be  placed  on  training  for  research,  development, 
diffusion,  or  evaluation. 

The  AERA  placement  service  data  (Goodwin  and  Worthen,  1970)  provide 
some  data  which  suggest  that  the  RDDE  market  (at  least  during  1968-70)  was 
glutted,  with  applicants  far  outnumbering  vacancies  in  educational  RDDE. 
However,  these  data  are  limited  by  at  least  three  factors:    (a)  the  locale  of 
the  annual  conventions  may  have  been  a  significant  factor  in  determining  who 
"iised  the  employment  services,  (b)  there  is  no  way  of  knowing  whether  the  pro- 
portion of  vacancies  and  applicants  registered  with  the  service  is  representa- 
tive of  the  comparable  national  proportions  in  the  educational  RDDE  market,  and 
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(c)  even  if  (a)  and  (b)  above  were  resolved,  the  data  represent  a  snap- 
shot In  time  which  provides  little  guidance  to  the  person  developing 
training  programs  for  the  future. 

The  several  studies  of  perceived  manpower  needs  In  various  regions 
suggest  that  many  persons  see  a  need  for  considerably  more  RDDE  personnel 
than  are  employed  In  educational  agencies  at  present.    However,  lack  of 
clarity  as  to  how  data  were  collected  or  estimates  were  made  and  differing 
definitions  of  educational  RDDE  render  further  synthesis  of  these  data 
hazardous  if  not  impossible.    In  the  one  survey  (Bargar  and  Hagan,  1970) 
where  respondents  were  asked  whether  or  not  they  actually  thought  addi- 
tional positions  for  RDDE  persons  would  or  could  be  provided  (as  opposed 
to  whether  such  positions  would  be  helpful),  the  results  were  anything  but 
optimistic.   This  creates  some  question  as  to  whether  similar  attrition 
would  result  in  the  other  "need-based"  estimates  if  reality  checks  of  this 
type  were  imposed  on  those  estimates. 

One  thing  which  might  be  said  with  some  degree  of  certainty  (although 
not  specifically  as  a  result  of  data  presented  herein)  is  that  future  man- 
power needs  will  be  tied  much  more  directly  to  funding  levels  for  educa- 
tional RDDE  than  to  educationists'  perceptions  of  how  many  RDDE  personnel 
are  needed.    In  other  words,  funds  will  determine  demand  for  RDDE  personnel 
in  education.    At  the  present  time,  it  seems  doubtful  that  there  will  be  a 
marked  change  in  the  near  future  in  the  amount  of  money  available  for  educa- 
tional RDDE.    If  the  proposed  National  Institute  for  Education  does  come 
into  existence,  then  a    rapid  infusion  of  funds  Into  the  research  field  is 
likely.    However,  as  with  the  passage  of  the  ESEA,  creation  of  the  NIE  ms^y 
result  in  a  large  amount  of  money  being  available  for  two  or  three  years. 
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but  then,  if  it  does  not  appear  that  the  NIE  is  accomplishing  its  goals, 
funds  msy  be  cut  of f  partially  or  even  completely.   Thus,  based  on  past 
experience  with  the  Monte  Carlo  characteristics  of  Federal  funding  of 
educational  RDDE,  it  is  most  difficult  to  project  likely  funding  levels 
for  RDDE  activities. 

In  summary,  the  current  data  on  future  supply  and  demand  of  educa- 
tional RDDE  personnel  are  simply  much  too  shaky  to  support  any  firm 
projections;  indeed,  they  defy  any  meaningful  synthesis.    Carefully  con- 
trolled studies  of  educational  RDDE  manpower  needs  and  supply  are  neces- 
sary before  any  real  direction  can  be  provided  to  trainers  of  RDDE 
personnel .^^    In  the  absence  of  such  studies,  trainers  of  necessity  will 
have  to  combine  their  best  interpretations  of  data  such  as  those  discussed 
in  this  section  with  judicious  use  of  the  crystal  ball  to  determine  how 
raar\y  and  what  type  of  personnel  should  be  trained  in  educational  RDDE. 


Paul  Hood,  of  the  Far  West  Regional  Educational  Laboratory,  is 
currently  designing  a  study  of  educational  RDDE  personnel  which  promises 
to  yield  much  valuable  information  in  this  area.  _ 
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CHAPTER  VIII 

A  SYNTHESIS  OF  THE  RESULTS  OF  RESEARCH  ON  THE  TRAINING 
OF  RESEARCH  AND  RE  SEARCH -RELATED  PERSONNEL  IN  EDUCATION 
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A  SYNTHESIS  0>  THE  RESULTS  OF  RESEARCH  ON  THE  TRAINING 
OF  RESEARCH  AND  RESEARCH-RELATED  PERSONNEL  IN  EDUCATION^ 

There  is  an  Increasing  number  of  research  Investigations  dealing 
with  the  training  of  personnel  In  educational  research*  development*  diffu- 
sion and  evaluation  (RDDE).   To  the  present  time*  however*  the  studies  have 
been  unrelated  In  any  progranmatlc  sense  and  the  results  appear  to  be 
fragmented  and  collectively  Impotent  In  guiding  trainers  of  RDDE  personnel. 
In  formulating  Its  objectives  for  the  current  project*  the  AERA  Task  Force 
on  Research  Training  was  convinced  that  the  results  of  studies  relating  to 
similar  variables  Important  In  RDDE  training  could  be  Integrated  Into  more 
applicable  knowledge  than  would  a^y  of  the  studies  viewed  Individually. 
The  Task  Force  therefore  set  as  one  of  Its  objectives  to  ".  .  .  synthesize 
extant  studies  of  educational  RDDE  personnel  or  studies  of  variables  directly 
related  to  training  such  persons"  (Objective  6).    Generalizations  drawn 
across  studies  might  well  offer  valuable  Insights  Into  the  training  process 
and  lead  to  the  formulation  of  exemplary  training  programs. 

In  order  to  draw  such  generalizations*  It  would  be  necessary  first 
to  Identify  relevant  studies*  to  examine  them  for  common  themes*  and  to 
synthesize  their  results  and  the  conclusions  drawn  from  them.   Such  a  pro- 
cedure Is  perilous,  however.  If  there  Is  doubt  about  the  truth  or  accuracy 
of  the  findings  or  conclusions  of  any  one  studly  which  contributes  to  the 
synthesis.   The  credibility  of  each  stu4y  therefore  must  be  established 
before  Its  results  can  be  Included  In  the  synthesis. 

Hhe  guidance  of  Dr.  William  J.  Gephart  In  the  effort  reported  In 
this  chapter  Is  gratefully  acknowledged.    In  particular*  his  assistance 
In  synthesizing  judgments  of  the  studies  and  In  deciding  which  studies  should 
be  Included  In  the  synthesis  was  of  great  value. 
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The  purpose  of  objective  6  was  to  collect  and  organize  research 
studies  having  direct  relevance  to  the  training  of  personnel  In  educational 
RDDE,  to  subject  the  relevant  studies  to  tests  of  methodological  adequacy 
by  expert  judges  of  research  quality,  and  to  Integrate  the  findings  and 
conclusions  of  those  studies  which  stood  the  test.    The  procedures  followed 
to  Identify,  describe,  and  judge  the  relevant  research  studies  are  outlined 
In  the  following  section.    The  procedural  description  Is  followed  by  a 
synthesis  of  the  results  of  those  studies  which  were  judged  to  be  niethodo- 
logically  sound. 

PROCEDURES 

The  procedures  followed  In  the  review  and  critique  of  research  are 
discussed  under  the  following  headings: 

(1)  Identification  and  collection  of  research  studies. 

(2)  Determination  of  the  relevance  of  each  study  to  RDDE  training 
variables. 

(3)  Description  of  the  relevant  studies. 

(4)  Development  of  cnteria  for  testing  methodological  adequacy. 

(5)  Judgment  of  the  relevant  studies. 
Identification  and  Collection  of  Studies 

Standard  library  search  procedures  were  utilized  by  the  Task  Force 
staff.   A  list  of  key  words  was  prepared  with  the  intent  of  setting  the 
broadest  possible  limits  for  the  identification  of  relevant  studies. 
Psychological  Abstracts ,  Sociological  Abstracts,  the  Education  Index,  Current 

2 

The  actual  abstracts  of  relevant  studies  and  the  specific  conclusions 
reached  during  the  judgment  process  were  originally  presented  in  Technical 
Paper  No.  27;  they  are  not  included  here.    The  research  studies  which  are 
included  in  the  synthesis  section  of  this  chapter  are  listed  in  Appendix  0 
to  this  report. 
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Index  to  Journals  in  Education,  and  Research  Studies  in  Education  (keyed 
to  Dissertation  Abstract  International)  were  reviewed  for  titles  and 
abstracts  that  had  potential  relevance  for  this  review.   The  Science 
Information  Exchange,  as  well  as  the  ERIC  system,  were  also  utilized.  The 
bibliographies  from  studies  known  to  be  important  works  in  the  area  were 
obtained  and  reviewed  for  any  additional  titles  that  should  be  included. 

Approximately  275  titles  were  identified.   Of  these,  approximately 
150  articles,  books,  and  manuscripts  were  discarded  because  they  were  not 
data-based  studies  or  because  the  content  was  subsequently  determined  to 
be  unrelated  to  RDDE  training. 

The  remaining  125  studies  were  requested  from  authors,  or  obtained 
from  the  University  of  Colorado  library  system  or  from  University  Microfilms. 
There  were  15  potentially  relevant  studies  which  were  not  obtained  due  to 
the  failure  of  the  author  to  respond  in  the  required  time  or  the  lack  of 
availability  of  the  works  from  the  library  system  or  the  inter-library  loan 
service.    (Example  of  potentially  relevant  studies  not  collected  were 
Chamberlain,  1971 ;  Schalock,  1970;  Davis,  1962;  and  DiLorenzo,  1967.)  The 
remaining  studies  were  read,  described  briefly  and  subjected  to  analysis 
by  staff  members  according  to  the  inclusion  criteria  developed  for  this 
project. 

Determination  of  Relevance 

Each  stucly  identified  was  subjected  to  the  following  criteria: 

1.       Studies  must  show  some  evidence  of  data  collection  and  reporting. 

The  sizable  number  of  position  papers  identified  were  thus  excluded 
from  consideration.   Examples  of  such  position  papers  are  Stanley 
(1966)  and  Kerlinger  (1965).    Exceptions  to  this  criterion  were  made 
for  works  by  Gideonse  (1969)  and  Bargar  (1967). 
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2.  Program  evaluations  are  excluded.   The  author  must  Intend  that 
his  findings  be  generallzable  beyond  a  single  program.  Because 
of  this  criterion,  evaluations  of  ESEA  Title  IV  training  programs 
were  excluded  from  consideration, 

3.  Studies  must  be  relevant  to  variables  related  to  training  of  RDDE 
personnel.   A  system  of  categories  was  developed  which  permitted 
each  study  to  be  classified  according  to  Its  degree  of  relevance 
for  this  project.   The  four  categories  are  discussed  below. 

Category  1 :    Studies  relating  specifically  to  the  training  of 
RDDE  personnel  in  education  formed  the  most  directly  relevant  group. 
Examples  of  studies  falling  into  this  category  are  Worthen  and 
Roaden  (1971),  Fleury  (1968),  Fields  (1971),  and  Sieber  (1968). 

Category  2:    The  next  most  relevant  group  consisted  of  those 
studies  investigating  training  variables  of  RDDE  personnel  in  the 
related  areas  of  the  social  and  behavioral  sciences.    Exampl es 
fitting  in  this  group  are  Clark  (1957) ,  Sibley  (1963),  and  Fiedler 
(1968). 

Category  3:   The  third  category  delineated  by  the  staff  consisted 
of  studies  of  educational  RDDE  personnel  themselves  (not  of  their 
training  ger  se)  including  manpower  studies  and  investigation  of  the 
setting  in  which  educational  RDD  and  E  takes  place.    Examples  here 
are  Persell  (1970),  Sieber  and  Lazarsfeld  (1966),  and  Clark  and 
Hopkins  (1969). 

Category  4:   This  category  was  made  up  of  studies  investigating 
researchers  in  related  areas.    Parts  of  Clark  (1957)  and  Sibley  (1963) 
fall  into  this  group. 
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Outside  the  core  of  studies  identified  in  (3)  was  a  large  number  of 
books  and  reports,  some  parts  of  which  are  relevant  to  RDDE  train- 
ing variables.   The  decision  to  include  or  exclude  each  document 
was  made  by  consensus  of  three  staff  members  based  on  an  extensive 
reading  and  summarizing  of  the  document  by  one  of  the  staff.  The 
decision  criteria  concerned  three  general  areas: 

a.  Information  in  the  study  must  be  sufficiently  specific  to 
RDDE  personnel:  i.e.,  it  must  break  down  categories  of 
"educational  personnel"  into  more  specific  categories  such 
as  "educational  research  personnel."   Studies  dealing  with 
graduate  students  in  all  fields  of  education  (with  sub-area 
unspecified),  such  as  Kidd  (1959),  were  thus  excluded.  The 
same  distinction  was  adhered  to  in  considering  studies  of 
personnel  in  related  fields. 

b.  The  variables  studied  must  be  directly  related  to  training, 
or  inferences  must  be  fairly  direct  between  the  variable 
studied  and  RDDE  training.  There  must  be  some  possibility  of 
control  over  the  training  variable  in  order  for  that  variable 
to  be  considered  for  inclusion.    Studies  which  dealt  with 
creativity,  such  as  Torrance  (1964),  as  well  as  "environmental 
press"  studies  such  as  Thistlethwaite  (1963),  were  thus 
excluded. 

c.  The  proportion  of  relevant  findings  in  the  report  to  total 
number  of  findings  must  be  sufficient  to  warrant  the  sizeable 
commitment  of  resources  involved  in  describing  and  critiquing 
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the  total  study.    That  is,  if  there  was  only  one  relevant 
finding  in  a  group  of  38  findings  in  a  large  study,  it  was 
decided  that  the  adequacy  of  the  entire  study  would  iiot  be 
assessed  just  to  determine  whether  or  not  that  one  finding  should 
be  included. 

The  rationale  for  these  criteria  relates  to  the  need  to  deal  in  depth 
with  the  most  directly  pertinent  studies  of  training  variables  and  to  scr^e'en 
out  from  the  "knowledge  base"  those  studies  which  are  only  tangentially 
related. 

Description  of  Studies 

Those  studies  which  satisfied  the  inclusion  criteria  were  assigned 
to  Task  Force  staff  members  for  description.   These  readers  were  either 
graduate  students  or  faculty  at  the  University  of  Colorado  who  designated 
research  as  a  major  or  minor  field  of  study.   They  were  oriented  to  the 
purposes  of  this  objective  and  trained  in  the  use  of  the  descriptive 
outline  which  had  been  developed. 

The  outline  for  description  of  each  study  included  the  problem  state- 
ment, the  statement  of  objectives,  and  the  hypotheses  or  questions  posed  by 
the  author.    Data  generation  was  described  according  to  the  population  studied, 
the  sample,  the  variables  under  study,  the  instruments  or  techniques  used 
to  collect  data,  the  percentage  of  survey  response  (where  appropriate)  and 
descriptions  of  treatment  variables  (in  the  case  of  experimental  or  quasi- 
experimental  designs).    Data  analysis  was  described  in  terms  of  the  type  of 
data  generated  for  each  variable  (nominal,  ordinal,  interval  or  ratio),  the 
statistical  technique  employed  for  each  variable,  and  indications  of  errors 
in  data  reduction  or  analysis.   A  list  of  the  author's  findings  was  presented. 
These  were  defined  as  narrative  statements  of  numeric  outcomes  and  were 
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accompanied  by  any  indications  found  by  the  reader  of  inconsistencies 
between  numeric  and  narrative  productions.  The  conclusions  reached  by 
the  author  were  presented  last.    Conclusions  were  defined  for  the  staff 
members  as  those  statements  by  the  authors  that  interpreted  or  explained 
the  findings  of  the  stuc^y*  as  well  as  the  generalizations  drawn  by  the 
author.    Some  problems  were  encountered  with  respect  to  the  conclusions 
since  the  above  definition  was  not  always  consistent  with  those  of  the 
authors.    Not  all  authors  listed  conclusions.     Some  so-called  conclusions 
were  simply  restatements  of  findings,  while  others  seemed  to  come  more  from 
the  author's  philosophical  position  than  from  the  data  he  gathered.  The 
readers  were  Instructed  to  accept  the  conclusions  which  were  listed  as  such  by 
the  authors.    In  cases  where  no  conclusions  were  explicitly  stated,  the 
readers  were  Instructed  to  list  those  statements  from  the  text  which  seemed 
to  explain.  Interpret,  or  draw  generalizations  from  the  findings. 

Development  of  Criteria  for  Testing  Methodological  Adequacy 

After  a  series  of  discussions  among  the  Task  Force  and  Task  Force 
staff  and  Investigations  of  various  methods  of  evaluating  research 
studies,  the  decision  was  made  to  use  the  methodology  proposed  by 
Gephart  (1969)  In  "Profiling  Instructional  Package,"  a  Phi  Dalta  Kappa 
occasional  paper.   The  method  proposed  consists  of  the  profile  of  the 
various  "elements"  of  the  research  process:   the  "Inherent  logical 
argument"  of  the  Investigation,  the  "selection  of  subjects  to  be  studied, 
structuring  of  experiences  for  those  subjects,  measurement  related  to 
those  experiences,  and  analysis  of  the  collected  data."   Gephart  makes 
the  assumption  that  each  element  of  a  stuify  must  be  evaluated  separately 
since  the  methodology  may  be  sound  in  some  areas  and  weak  in  others. 
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Modification  of  the  Gephart  system  was  necessary  for  several 
reasons.    First  it  was  important  to  emphasize  those  aspects  of  the 
evaluation  which  pertained  to  survey  research,  since  this  methodology 
has  been  most  often  used  by  authors  of  research  training  studies. 
Second,  the  reasons  for  determining  the  methodological  adequacy  of  these 
studies  (i.e.,  deciding  whether  or  not  the  study  would  be  included  in 
the  synthesis  of  results)  was  idiosyncratic  to  this  project  and  allowed 
de-emphasis  of  the  conceptual,  pre-data  collection  phases  of  the  study. 
Last,  through  a  series  of  discussions  with  experts  in  the  field,  it  was 
found  that  evaluating  the  quality  of  research  is  often  a  subjective  pro- 
cess in  which  the  research  methodologist  brings  to  bear  his  accumulated 
knowledge  and  experience  in  order  to  arrive  at  an  overall  decision  about 
the  adequacy  of  the  study.   Therefore,  it  was  decided  that  the  profile 
of  elements  in  the  evaluation  system  should  supplement  rather  than  replace 
this  process. 

On  the  basis  of  these  discussions  two  types  of  judgment  format  were 
prepared,  one  for  status  studies  (normative,  survey,  correlational)  and 
another  for  experimental  (and  quasi -experimental )  studies.   The  elements 
of  the  format  for  status  studies  are  given  below  along  with  the  general 
evaluative  criteria.    (No  studies  which  were  classified  as  experimental 
or  quasi -experimental  were  identified  as  relevant  for  this  review.) 

1.  The  Popul ation  of  the  study  -  (clear  definition  of  the  population 
and  its  appropriateness  to  the  problem  investigated) 

2.  The  Sample  studied  -  (the  definition  and  the  size  of  the  sample  as 
well  as  the  representativeness  of  the  sample  to  the  population) 
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3.       The  Variables  studied  -  (clear  delineation  and  operational 

definition  of  the  variables  as  well  as  the  appropriateness  of 


4.  The  Instrument  or  technique  used  for  data  collection  -  (the 
sufficiency  of  the  instrument  in  terms  of  objectivity,  reliability 
and  validity) 

5.  Percentage  of  Response  in  survey  studies  -  (the  sufficiency  of 
the  return  rate  and  the  adequacy  of  non-response  bias  checks) 

6.  Data  Analysis  -  (the  appropriateness  of  the  statistical  techniques 
applied  to  each  type  of  data  as  well  as  indications  of  errors  in 
analysis  that  would  alter  the  findings  significantly)  • 

7.  Findings  -  (the  accuracy  of. the  translation  from  numeric  outcome 
to  narrative  presentation;  the  determination  of  whether  or  not  the 
findings  were  unequivocal) 

8.  Conclusions  -  (the  determination  of  whether  or  not  each  conclusion 
is  a  logical  and  justifiable  interpretation  or  explanation  of  the 
data  and  whether  or  not  the  conclusion  is  warranted  by  the  findings 
of  the  study) 

9.  General  Coninents  and  Recommendations  of  the  judges  regarding  inclu- 
sion or  exclusion  of  the  study  for  the  synthesis  of  results. 

In  addition  to  these  elements,  several  critical  points  were  identi- 
fied for  the  judges.    If  the  stu4y  failed  at  any  of  these  termination 
points,  the  judge  was  requested  to  terminate  his  evaluation.  These 
termination  points  were  the  following: 


the  variables  to  the  problem  investigated) 
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a.  if  the  sample  of  the  stucly  was  both  unspecified  and  undefined, 
so  that  the  reader  could  not  determine  who  was  being  studied, 
exclude  the  stuc(y; 

b.  if  all  data  were  analyzed  by  an  inappropriate  statistical 
technique,  exclude  the  stuc|y; 

c.  if  errors  in  analysis  were  serious  enough  to  alter  all  the 
findings  significantly,  exclude  the  stucjy,  and; 

d.  if  an  accumulation  of  small  problems  or  overall  weaknesses 
caused  the  judge  to  doubt  seriously  the  credibility  of  the 
findings,  exclude  the  stuc|y. 

Judgment  of  the  Studies 

The  staff  member  who  read  and  described  a  particular  study  was  also 
the  first  judge  for  that  stu4y.   Staff  members  were  oriented  to  the  pur- 
poses of  this  phase  of  the  objective  and  trained  in  the  use  of  the  judgment 
format. 

The  outside  experts  who  served  as  second  judges  on  the  studies  all 
hold  degrees  in  educational  research  or  research -related  areas  and  include 
nationally  recognized  experts  in  research  methodology  as  well  as  several 
recent  graduates  of  the  University  of  Colorado,  Laboratory  of  Educational 
Research  (ESEA  Title  IV  training  program),  where  the  training  emphasis  was 
on  research  methodology.^  These  outside  judges  were  provided  with  the 
description  of  the  stuc<y  and  (in  most  cases)  the  document  itself,  along  with 
orientation  to  the  use  of  the  judgment  format  and  the  purposes  of  this  phase 
of  the  study. 


ri  outside^ udges  were  Nancy  W.  Burton,  James  R.  Collins,  Gene  V 

Glass,  William  L.  Goodwin,  Arlen  R.  Gullickson,  Kenneth  D.  Hopkins,  Stephen 
G.  Jurs,  Perc  Peckham,  W.  Todd  Rogers,  James  R.  Samiers,  and  Douglas  D. 
Sjogren.  I'V- 
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In  a  few  cases  there  were  disagreements  between  the  first  and 
second  judges'  recommendations  concerning  the  inclusion  or  exclusion  of 
a  stu4y.   To  resolve  the  disagreement,  an  expert  in  research  and  evalua- 
tion from  the  Colorado  State  Department  of  Education  was  called  upon  to 
act  as  a  third  judge  to  resolve  differences  between  the  first  two.* 

Finally,  a  member  of  the  AERA  Task  Force  synthesized  the  judgments 
made,  summarized  the  judges'  comments  on  the  studies,  and  aided  in  the 

process  of  deciding  whether  the  findings  and/or  the  conclusions  of  each 

5 

stu4y  should  be  included  in  the  synthesis  of  results  which  follows. 

SYNTHESIS 

Although  the  Task  Force  has  been  concerned  with  the  training  of  all 
types  of  educational  RDDE  personnel,  this  synthesis  deals  almost  exclusively 
with  the  training  of  researchers  per  se  rather  than  the  entire  range  of 
research,  development,  diffusion,  and  evaluation  personnel.    This  is  not  a 
denigration  of  the  other  roles;  rather,  it  is  a  reflection  of  the  focus  of 
existing  studies  on  the  training  of  educational  research  and  research- related 
personnel . 

The  knowledge  base  provided  by  research  on  the  training  of  educational 
researchers  has  antecedents  in  several  areas  of  research  currently  being 
pursued.   Among  these  are  studies  of  research  quality,  studies  of  research 
methodology,  and  manpower  studies.  A  brief  summary  of  the  nature  of  research 
in  these  areas  is  given  below,  but  individual  studies  will  not  be  summarized. 

Primary  interest  in  this  chapter  is  devoted  to  the  area  of  research 
impinging  directly  on  the  training  of  educational  research  personnel.  Studies 
in  this  area  were  subjected  to  the  tests  of  methodological  adequacy  described 

4 

This  judge  was  John  P.  Ahlenius;  .who  received  his  training  at  CIRCE, 
University  of  Illinois.  ^  . 

William  J.  Gephart  served  in  this  capacity.  f^if^ 
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above,  and  those  which  withstood  the  test  are  included  in  the  present 
synthesis.   The  synthesis  is  presented  in  two  sections,  one  which  deals 
with  researchers  and  the  research  setting,  and  one  which  deals  with  the 
research  training  process  itself.   The  two  sections  of  the  synthesis 
immediately  follow  the  summary  of  the  nature  of  research  in  the  background 
areas . 

The  Nature  of  Research  in  the  Background  Areas 

Studies  of  Research  Quality 

Broadly  pictured,  systematic  studies  of  the  training  of  researchers 
stem  from  concerns  about  the  quality  of  the  research  effort.  Published 
statements  expressing  this  concern  date  back  to  1923  (Brooks,  1923)  and  can 
be  found  with  increasing  frequency  over  the  ensuing  years.    The  Brooks 
article  presents  a  set  of  criteria  to  be  used  in  considering  educational 
research  and  represents  an  attempt  to  improve  educational  research  by 
providing  guidelines  for  use  in  designing  research  efforts.    A  slightly 
different  approach  is  the  analysis  of  the  errors  made  in  completed  research. 
The  earliest  example  of  this  is  Wilson's  (1934)  effort.   The  purpose  of 
studies  in  this  category  is  to  make  recommendations  on  what  research  ought 
to  be,  based  on  a  study  of  errors  in  prior  research  efforts. 

The  product  of  this  line  of  inquiry  is  improved  understanding  of  the 
general  nature  of  the  research  process,  and  in  turn  the  processes  by  which 
researchers  ought  to  be  trained.    It  has  led  to  work  by  Gephart  and  Bartos 
'(1969)  who  proposed  that  there  exist  in  any  piece  of  research  five  facets: 
ah  inherent  logical  argument,  a  representativeness  facet,  a  measurement 
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facet,  a  treatment  facet,' and  an  analysis  facet.   The  authors  propose  that 
assessment  of  these  facets  is  necessary  to  determine  the  validity  of  the 
research  and  to  determine  the  confidence  that  can  be  placed  in  its  conclusions. 
Studies  of  Research  Methodology 

The  second  line  of  antecedent  research  deals  with  the  study  of 
specific  research  methods  and  techniques  such  as  the  validity  of  a  measur- 
ing instrument,  the  role  of  the  "Hawthorne  Effect,"  the  role  of  Bayesian 
statistics,  etc.   This  area  of  inquiry  is  one  on  which  the  training  of 
researchers  must  depend,  for  it  provides  the  specific  content  for  the 
Instructional  program. 

Although  studies  of  both  research  quality  and  research  methodology 
are  Important  antecedents  of  the  studies  of  research  training,  they  ^o  not 
deal  with  the  training  of  research  and  research-related  personnel  per  se. 
No  further  effort  has  been  made  to  state  in  this  synthesis  what  knowledge 
has  been  produced  by  these  studies. 

Manpower  Studies 


reported  In  Technical  Paper  No.  27.  A  synthesis  of  the  results  of  these 
studies  Is  presented  in  Chapter  VII  above. 


An  example  of  this  type  of  assessment  is  the  approach  used  in 
determining  the  methodological  adequacy  of  studies  of  research  training 


Studies  which  deal  with  the  supply  of  and  demand  for  personnel  with 
capabilities  in  educational  RDDE  were  included  in  the  review  of  research 


contained  herein. 
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Studies  of  Researchers  and  the  Settings 
in  which  Research  is  Conducted 

Educational  research  is  done  by  people,  individually  and  collectively, 
in  an  environment.   The  nature  of  the  individuals  and  the  physical  and 
socio-emotional  character  of  the  environment  affect  the  nature  of  the  work 
done.   Research  in  these  areas  attempts  to  identify  the  educational 
researchers,  to  describe  them  on  selected  variables,  to  characterize 
the  institutions  in  which  they  work,  and  to  assess  their  productivity.  In 
assessing  productivity,  some  information  directly  related  to  the  effective- 
ness of  training  is  produced.   Research  on  the  other  factors  described 
above  provides  related  and  supportive  information  but  not  direct  statements 
about  research  training. 

The  values  of  educational  researchers,  their  educational  background, 
their  employment  positions,  and  the  type  of  work  they  do  are  all  ^jmportant 
in  understanding  both  the  nature  of  the  task  for  which  researchers  are 
being  prepared  and  the  means  by  which  the  desired  expertise  can  be  acquired. 
This  does  not  imply  that  the  goal  of  educational  research  training  programs 
should  be  simply  to  reproduce  the  existing  type  of  researchers  for  work 
within  the  existing  institutional  settings  in  which  research  is  currently 
conducted.    Rather,  it  is  information  which  should  be  relevant  as  future 
training  alternatives  are  delineated  and  assessed. 

A  notable  problem  was  encountered  in  the  attempt  to  synthesize 
findings  from  studies  which  describe  existing  researchers.    Although  the 
population  purportedly  was  the  same  across  the  studies,  different  authors 
identified  their  groups  differently.   Buswell,  et  al.  (1966),  for  example, 
studied  those  persons  who  had  received  their  doctorate   in  1954  and  1964 
from  schools  or  departments  of  education.    Other  authors  identified  their 
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samples  differently,  sometime'S  including  researchers  from  substantive 
fields  outside  education  (Clark, ''s 957;  Bargar,  et  al.,  1965)7  Some 
identified  their  samples  by  consulting  directories  of  relevant  professional 
associations  (Worthen  and  Roaden,  1968)  or  by  consulting  the  National 
Register  of  Educational  Researchers   as  Worthen  (1969)  did.   Still  others 
used  a  combination  of  these  methods  (Bargar,  et  al.,  1965).   When  the 
differences  in  sample  identification  are  pertinent  in  understanding  differ- 
ences in  the  results  of  studies  discussed  in  this  chapter,  they  will  be 
re-emphasized. 

Demographic  and  Personal ogical  Variables 

The  following  demographic  and  personal  ogical  variables  related  to 
researchers  have  been  studied:   age,  sex,  geographic  location,  religion, 
socio-economic  status,  and  personality  traits.    A  synthesis  of  the  methodo- 
logically adequate  studies  is  given  in  the  paragraphs  below. 

Age.   The  average  age  of  the  researchers  studied  by  Bargar,  et  al . 
(1965)  was  45  years  while  the  average  age  of  Persell's  (1971)  group  was 
41  years.   The  juxtaposition  of  these  results  gives  the  impression  that 
the  average  age  of  the  research  group  is  decreasing  over  time.    There  are 
alternative  explanations  for  this  difference,  however,  one^of  which  is  the 
difference  in  defining  the  population  employed  by  the  two  authors.  This 
could  be  the  case  since  Re rsell 's  group,  by  definition,  was  composed  of 
productive  scholars  (authors  of  research  reports),  and  (as  noted  below) 
such  persons  tend  to  be  younger  upon  receipt  of  degree  than  researchers  at 
large,  which  is  more  representative  of  the  group  Bargar  studied.  These 

^Persell  (1971)  documented  the  overlap  between  psychology  and 
education  in  finding  that  psychology- influenced  educational  research  in 
both  research  methods  empl  oyed  and  the  probl  ems  sel  ected  foir  i  n vesti  gati  on . 
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findings  relate  to  the  age  of  researchers  at  the  time  they  responded  to 
the  respective  surveys.   A  related  factor  —  the  age  at  which  the  researchers 
received  their  doctorates  —  has  been  studied  in  several  projects  and  is 
discussed  below. 

Early  entry  into  a  research  career  appears  to  be  positively  related 
to  productivity.   Buswell,  et  al .  (1966)  concluded  that  those  who  received 
a  doctorate  by  age  32  are  more  likely  to  be  productive  than  those  who 
received  it  at  age  40  or  older.    They  found  further  that  those  persons  whom 
they  identified  as  the  most  productive  scholars®  tended  to  have  completed 
their  doctoral  degree  before  age  30.^   This  is  consistent  with  Clark's 
(1957)  conclusion  that  "significant  contributors"  to  the  field  of  psychology 
emerge  early  in  their  careers.    Persell  (1971)  found  that  a  greater 


^wo  different  populations  are  referred  to  in  the  Buswell,  et  al. 
(1966)  stu<ly.  The  first  consisted  of  all  those  persons  who  receiveS 
doctorates  in  education  in  1954.   These  were  divided  into  a  productive 
(two  or  more  research  articles  published  in  ten  years)  and  a  non-productive 
group.    The  second  population  was  studied  by  Heiss  and  consisted  of  those 
tliirty-one  scholars  who  had  contributed  the  greatest  nuraber^o'f  research 
articles  in  journals  related  to  education  and  were  selected  on  the  basis 
of  their  judged  contribution  to  the  field.  These  will  be  referred  to 
here  as  the  "most  producti  ye"  researche rs . 

,  j  ;i  for  consideration  of 

pixjducti  vlty  of  researdiers.  ' 
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percentage  of  researchers  under  the  age  of  thirty  were  doing  good  quality 

research  than  was  true  for  researchers  aged  50  or  older 

In  addition  to  receiving  the  doctoral  degree  at  a  relatively  early 
age,  the  most  productive  scholars  publish  research  soon  after  their  entry 
into  research  training.   Buswell,  et^  al..  (1966)  found  that  productive 
researchers  tended  to  have  published  research  before  or  within  one  year  of 
the  completion  of  the  doctorate.       More  than  half  of  those  identified  as 
the  most  productive  researchers  had  published  a  research  article  or 
presented  a  paper  at  a  professional  meeting  before  completing  the  doctorate. 

The  collected  findings  appear  to  point  in  the  same  direction  --  that 
those  who  receive  their  degree  at  a  younger  age  are  more  likely  to  make 
contributions  to  the  field  through  research.    At  the  same'-time,  other 
results  indicate  that,  on  the  average,  education  doctorates  receive  the 
doctoral  degree  relatively  late.   Wilder  (1966)  found  that  the  median 

•  interval  between  the  bachelor's  degree  and  the  doctorate  for 


^®The  reader  is  alerted  to  the  fact  that  Persell's  definition  of 
research  quality  is  somewhat  different  from  many  previous  definitions. 
Typically,  research  quality  has  been  operationally  defined  as  the  adequacy 
of  the  research  methods  and  techniques  (e.g.,  design,  analysis)  used  by 
the  investigator  to  produce  unequivocal  data.    Such  definitions  of  research 
quality  result  in  the  primary  focus  of  assessment  of  quality  being  on 
freedom  from  methodological  flaws  and,  consequently,  on  how  much  confidence 
can  be  placed  in  the  results  of  a  study.    Persell  asked  expert  judges  to 
give  three  ratings  to  a  piece  of  research  based  on  its  (a)  contribution 
to  theory,  (b)  contribution  to  practice,  and  (c)  use  of  (or  contribution  to) 
research  methods.'  The  latter  category  overlaps  with  usual  w^s  of  judging 
research  quality,  while  the  two  additional  categories  add  the  dimension  of 
judging  quality  in  terms  of  theoretical  or  practi'cal  significance.  However, 
since  two-thirds  of  the  judges'  responsibility  was  focused  on  what  results 
from  a  study— i  .e. y  the  impact  it  has— it  ma^y  be  that  there  is  relatively 
TesFemphasis  on  judging  methodological  adequacy  than  is  true  with  narrower 
definitions  of  research  quality  where  this  is  the  sole  focus  of  the 
qualitative  judgment. 
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persons  in  the  field  of  reading  was  15  years.     Assuming  that  the  average 
age  of  persons  receiving  their  bachelor's  degree  was  21  or  22,  this  would 
indicate  that  half  of  education  doctorates  receive  their  doctorate   at  age 
35  or  older  J ^ 

The  discrepancy  between  the  age  at  which  most  potential  scholars  in 
the  field  receive  their  highest  degree  and  the  age   at  which  researchers 
are  most  likely  to  contribute  is  a  significant  matter.   The  inference  could 
be  made  that  education  doctorates,  being  older  at  receipt  of  degree  than 
persons  in  other  fields,  would  ipso  facto  be  less  productive.   The  relatively 
late  entry  of  educational  doctorates  into  their  careers  and  the  possible 
Interference  of  their  pre-doctoral  career  experiences  are  related  variables 
which  will  be  considered  together  later  in  this  chapter. 

Sex.    Bargar,  et  al..  (1965)  found  that,  of  all  persons  identifying 
themselves  as  part  of  the  educational  research  community,  the  large  majority 
are  males.    This  Is  consistent  with  Persell  's  (1971)  conclusion  that 
females  are  under-represented  among  researchers.   Clark  (1957)  found  that  only 
5  percent  of  the  "significant  contributors"  to  the  field  of  psychology  were 
females,  while  females  account  for  26  percent  of  the  membership  of  the  American 
Psychological  Association.   Wilder  (1966)  found  that  a  smaller  percentage 
of  female  "reading  experts"  b^ome  researchers  than  is  true  for  males. 

The  under-representatlon  of  females  among  doctorates  in  education 
and  related  areas  Is  consistent  with  their  representation  in  other  professions. 

^^Wllder  (1966)  studied  persons  identified  as  experts  in  the  field  of 
reading.   They  were  identified  as  experts  by  virtue  of  their  training  and 
degrees  received  in  that  field,  as  well  as  positions  held  or  research 
conducted.  They  represent  a  subset  of  those  with  advanced  degrees  in  education. 

^^A  further  question  would  be  whether  the  age  distribution  is  the 
same  now  as  it  was  In  196B. 
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In  contrast,  it  is  of  interest  to  consider  the  preponderance  of  females 
in  fields  of  educational  practice,  as  opposed  to  research.   The  issue  of 
sex  differences  in  personnel  needs  as  expressed  in  vocation  might  be  used 
to  explain  this  anomaly  of  sex  composition  of  the  educational  community. 
Differences  in  the  role  prescriptions,  role  demands,  and  role  stereotypes 
that  characterize  the  two  areas  of  work  must  also  be  considered.  Self- 
selection  is  another  plausible  alternative  explanation  of  the  sex 
I  composition  of  the  educational  research  community.   Conversely,  another 

plausible  explanation  is  that  the.  sex  differential  can  be  at  least  partially 
explained  by  systematic  exclusijon  of  females  from  the  research  community. 
In  the  absence  of  an  adequate  empirical  base,  however,  all  the  above  state- 
ments  must  be  viewed  as  conjecture  at  this  point. 

Persell  (1971)  found  a  relationship  between  research  quality 
and  sex  which  was  quite  different  from  the  relationship  between  sex  and 
^  research  productivity  just  described.    That  author  found  that  39  percent 

^  of  the  females  In  her  sample  are  doing  good  quality  research. 

I  whereas  this  was  true  for  only  29  percent  of  the  males  In  her  sample.  The 

I  discrepancy  between  quality  and  productivity  as  it  relates  to  sex  is  not 

necessarily  a  contradiction,  for  Persell  studied  a  population  which  was 
I  productive  (of  at  least  one  piece  of  empirical  research)  by  definition. 

In  another  part  of  the  same  study  Persell  found  no  relationship  between 
quantitative  productivity  and  overall  quality  of  research.  [ 

To  sunmarize  the  research  results  on  the  sex  variable,  it  can  be 
said  that  the  educational  research  community  is  predominantly  male,  with 


I 
I 
I 
I 
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the  male  researchers  also  more  likely  than  females  to  be  among  the  productive 
members  of  the  connunlty.   Considering  only  the  productive  persons,  however, 
the  female  researchers  are  making  proportionately  more  contributions  to 
theory,  educationaVpractice,  and  research  methods. 

Religion,  race,  socio-economic  status.    Persell  (1971)  concluded  that 

\ 

the  population  of  researchers  is  more  diverge  in  terms  of  religion  than  in 
sex  or  age.   She  found  that  the  proportion  of  the  researchers  who  are 
Protestant  or  Catholic  is  less  than  would  be  expected  on  the  basis  of  the 
proportions  of  Protestants  and  Catholics  in  the  population  as  a  whole.  The 
percentage  of  whites  among  the  researchers  was  97  percent.    Persell  also 
concluded  that  "the  socio-economic  status  of  the  researcher's  family  had  no 
striking  relationship  to  research  quality"  (Persell,  1971,  p.  228). 

Geographic  location.    In  examining  the  portion  of  the  country  from 
which  researchers  have  come,  Bargar,  et  al_.  (1965)  found  that  the  majority 
of  individuals  had  come  from  the  heavily  populated  northeast  and  north- 
central  states  extending  from  Missouri  and  Minnesota  to  New  Jersey  and 
Massachusetts.    In  her  study  of  researchers,  Persell  (1971)  concluded  that 
more  had  come  from  the  midwest  than  from  any  other  region.   These  two. 
regions,  of  course,  overlap  to  a  large  extent. 

Personality  traits.   Buswell^^^  al^.  (1966,  p.  87)  reported  that 
in  terms  of  development  of  research  interest,  the  most  productive  group  of 
scholars  "...  had  numerous  iarly  interests  of  an  inquisitive  nature  and 
had  pursued  these  interests  systematically  and  independently."   Their  range 
of  outside  Interests  was  broad.   They  were  characterized  as  having  a  high 
level  of  theoretical  orientation  and  autonomy  as  well  as  the  ability  to 
deal  with  complex  ideas.   Persell  (1971)  noted  that  research  quality 
is  relatied  to  Intel lisctual  orientation,  which  she  described 
.  •  .  ■    .    ..    .'.    .     '  .  .  •  ■•  .    ■   .  ^ 
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as  being  intellectually  curious  about  substantive  questions  in  a  scientific 
field. 

Academic  Training  and  Career  Experiences 

The  academic  training  and  career  experiences  of  researchers  have 
been  surveyed  in  several  studies.    Attempts  have  been  made  to  relate  these 
experiences  to  the  subsequent  contribution  of  the  researchers. 

Undergraduate  training.    About  half  of  the  researchers  studied  by 
Bargar,  et  al^.  (1965)  had  majordd  in  education  as  an  undergraduate.  As 
might  be  expected,  a  larger  proportion  (70  percent)  of  those  reading 
experts  studied  by  Wilder  (1966)  had  an  undergraduate  major  in  education. 
Half  of  the  significant  contributors  to  the  field  of  psychology  had  majored 
in  that  field  as  undergraduates  (Clark,  1957). 

Concerning  the  undergraduate  experiences  associated  with  subsequent 
productivity,  Buswell,  et  a^.  (1966)  found  that  productive  researchers^ 
(those  education  doctorates  who  had  published  two  or  more  research  articles 
In  ten  years)  were  more  likely  to  have  received  undergraduate  degrees  from 
institutions  which  have  doctoral  degree  programs.    In  comparing  education 
doctorates  who  are  productive  with  those  who  are  not,  only  one  difference 
in  undergraduate  major  was  found.   Those  who  majored  in  psychology  were 
more  likely  to  becom6  productive  of  educational  research  than  undergraduate 
majors  in  other  fields.   A  negative  relationship  was  found  between  tTie 
number  of  undergraduate  education  courses  taken  and  later  productivity. 
The  most  productive  researchers  (Heiss,  in  Buswell,  et  al_.    1966)  usually 
had  a  liberal  arts  undergraduate  major. 

It  would  be  possible  to  juxtapose  these  findings  and  conclude  that, 
while  educational  researchers  as  a  whole  are  likely  to  have  undergraduate 
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backgrounds  In  education,  those  who  turn  out  to  be  productive  are  from 
liberal  arts  programs.   Recruitment  of  potentially  more  productive 
personnel,  therefore,  should  be  aimed  away  from  education- trained  people. 
Such  a  conclusion  falls  to  consider  the  fact  that  the  productive  researchers 
studied  by  Heiss  were  a  highly  selected  group  of  scholars  both  In  education 
and  In  related  disciplines,  who  had  published  a  large  number  of  research 
articles  of  concern  to  education  and  were  selected  on  the  basis  of  their 
pre-eminence  In  the  field.   Although  one  should  not  Infer  that  all  educa- 
tional researchers  ought  to  be  drawn  from  liberal  arts  backgrounds.  It 
appears  that  those  persons  are  likely  to  achieve  pre-eminence  In  greater 
proportions  than  those  with  undergraduate  education  backgrounds.  Clark 
reported  that  of  his  "significant  contributors"  three  out  of  four  reported 
being  In  the  top  five  percent  academically  of  students  In  their  respective 
majors. 

'  Graduate  training.    In  their  study  of  all  persons  who  wished  to 
Identify  themselves  with  the  educational  research  community,  Bargar,  et  a],. 
(1965)  found  that  at  the  doctoral  level,  the  most  common  major  was  education; 
however,  education  surpassed  psychology  by  only  a  small  percentage.    It  was 
found  that  the  majority  of  respondents  did  not  receive  their  primary  train- 
ing In  research  methods  £er  se.  For  those  who  specified  a  subfleld  of 
their  major,  the  most  common  subflelds.  In  descending  order  of  frequency,  were: 
educational  administration,  educational  psychology,  and  educlatlonal  subject 
areas.   In  another  study.  Wilder  (1966)  found  that  92  percent  of  his 
reading  experts  received  their  doctorates  from  departments  of  education. 

The  relationship  between  advanced  degree  riiajor  and  research 
productivity  has  also  been  Investigated.   Buswell,  et  al.  (1966)  reported 
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that  graduate  education  majors  are  no  more  or  less  likely  to  become 
productive  researchers  than  persons  who  have  majored  In  any  other  field. 
According  to  Persell  (1971),  education  doctorates  are  just  as  likely  as 
behavioral  science  doctorates  to  be  productive,  providing  they  have  had 
research  experience  as  a  graduate  student J ^  Among  those  scholars  who 
are  oriented  to  research,  Persell  found  that  behavioral  science  doctorates 
are  more  likely  than  education  doctorates  to  be  contributing  to  theory, 
but  no  more  likely  to  be  contributing  to  educational  practice  or  to  the 
use  of  research  methods. 

The  specific  experiences  of  the  researchers  during  their  graduate 
training  have  also  been  investigated.   Those  relating  specifically  to 
research  apprenticeship  experiences  are  discussed  in  a  later  section;  all 
others  are  discussed  here. 

Buswell,  et  al[.  (1966)  concluded  that  working  for  a  doctorate  in 
education  was  largely  a  part-time  pursuit  while  holding  a  full-time  job, 
usually  in  schools.   They  also  found  that  productive  researchers  and  non- 
productive researchers  were  significantly  different  with  respect  to  the 
number  of  months  th^y  had  spent  in  continuous  full-time  residence,  with 
productive  researchers  spending  more  time  in  residence.   When  all  doctorates 
studied  were  viewed  together,  it  was  found  that  a  large  proportion 
(69  percent)  had.avlapse  of  six  years  or  more  between  their  initial  enroll- 
ment in  doctoral  studies  and  the  completion  of  their  doctoral  degree. 
This  phenomenon  is  not  limited  to  those  in  education.   Clark  (1957)  found 

'13^  ■  • , 

This  relationship  .is  not  surprising;  graduate  research  experience 
seems  likely  to  lead  to  later  [research  more  often  ,  than  non-research  exper- 
iences, regardless  of  departmental  affiliation  of  the  graduate  students. 
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that  for  the  "significant  contributors"  in  psychology,  the  median  lapse 
between  the  B.  A.  and  the  Ph.D.  was  four  years,  while  for 
psychologists  in  general  that  lapse  was  about  eleven  years. 

In  terms  of  the  institutions  where  graduate  training  was  conducted, 
information  from  Bargar,  et  al_.  (1965)  shows  that  at  the  time  of  their 
study  (1964)   state  universities  accounted  for  slightly  over  half  of  all 
highest  degrees  obtained.   Buswell,  et  al^.  (1966)  found  that  the  most 
productive  scholars  In  the  field  earned  doctoral  degrees  at  Harvard, 
Columbia,  Chicago,  Minnesota  or  Ohio  State. 


disciplines  and  Is  not  exclusively  the  job  of  education  departments. 
Education  departments  do  produce  a  greater  proportion  of  educational 
researchers  than  does  any  other  department.   No  unequivocal  support  can 
be  found  for  systematically  favoring  training  in  one  discipline  over 
another  In  effecting  greater  subsequent  research  productivity  (although 
there  Is  some  support  for  viewing  undergraduate  majors  in  psychology  as 
potentially  more  productive  of  research).   Of  those  who  have  been  trained 
in  education,  however,  relatively  fewer  approach  graduate  training 
ininedlately  after  receiving  their  bachelor's  degree  or  pursue  it  continu- 
ously, two  factors  which  are  positively  related  to  subsequent  productivity. 

The  type  of  doctorate.    Although  the  Ph.D.  is  often  described  as  a 
research  degree  and  the  Ed.D.  Identified  as  a  professional  degree,  the 
empirical  evidence  from  several  studies  dealing  with  the  actual  differences 
*  between  the  two  is  unclear.    For  example,  Bargar,  et  al^.  (1965)  found 
that  58  percent  of  his  self -designated  group  of  educational  researchers  had 


In  summarizing  these  findings,  it  appears  that  the  undergraduate 


researchers  Is  divided  among  the 
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the  Ph.D.  and  23  percent  had  the  Ed.D.   These  figures  were  not  cross- 
tabulated,  however,  with  the  field  or  department  in  which  the  degree  was 
obtained.    Therefore,  information  to  compare  the  proportion  of  Ed.D.s 
among  all  education  doctorates  with  their  proportion  in  the  research 
community  is  lacking.   One  can  draw  some  inferences  from  the  simple 
frequencies  presented  by  Bargar  and  his  ^colleagues.    They  report  that 
of  the  3216  persons  in  their  analysis  group  who  held  doctoral  degrees, 
915  held  the  Ed.D.  and  2296  held  the  Ph.D.    They  also  report  1432  doctoral 
degrees  in  education.   Although  no  cross-tabulation  is  provided,  it  is 
reasonable  to  assume  that  all  915  Ed.D.  degrees  are  held  by  persons  in 
education,  leaving  only  517  doctoral  degrees  of  other  types  —  probably 
almost  exclusively  Ph.D.  —  in  the  education  group.   Thus,  of  the 
education  doctorates  In  Bargar's  group  of  researchers,  holders  of  Ed.D. 
degrees  outnumber  holders  of  Ph.D.  degrees  by  almost  two  to  one,  raising 
questions  about  whether  the  Ph.D.  Is  really  more  of  a  research  degree 
than  the  Ed.D.    Without  knowing  the  relative  frequency  of  the  two  types 
of  degree  in  the  field  of  education  as  a  whole,  however,  the  frequencies 
in  Bargar's  group  cannot  be  Interpreted  meaningfully.    On  the  surface,  it 

appears  that  data  from  the  Buswell,  et  al^.  study  (1966)  could  be  used  to 

./  ■ 

establish  a  basis  for  comparison  of  proportions  of  degree  type    in  the 
field.   Comparison  with  the  proportions  in  Buswell.,  et  al^.  (1966)  Is 
difficult  since  certain  subspecialties  were  categorized  differently  by  the 
two  author's.    For  example,  Bargar,  et  al^.  (1965)  categorized  those  persons 
who  had  majored  in  educational  psychology  under  the  general  heading  qf*  \ 
psychology.   Many  of  these  same  persons  actually  received  doctorates  in 
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departments  of  education  and  would  have  been  classified  In  education  by 
Buswell,  et  aT[.(1966). 

Buswell  and  his  colleagues  concluded  that  differences  In 
the  type  of  research  methodology  employed  by  Ph.D.  and  Ed.D.  candidates 
In  conducting  their  dissertation  research  were  not  large..  In  this  study 
also,  the  biggest  differences  In  research  were  between  field  majors  rather 
than  types  of  degree.    In  terms  of  subflelds  within  education,  Buswell, 
et  al^.  (1966)  found  that  more  people  In  the  area  of  educational  psychology 
were  researchers  than  non-researchers,  while  for  persons  with  a  doctorate 
In  educational  administration  the  situation  was  reversed.   Other  subflelds 
of  education  showed  less  difference  In  the  percentages  of  persons  In  the 
research  and  non-research  groups. 

There  are  counter  examples  of  this  trend  to  minimal  differences 
among  holders  of  the  two  degree  types,  however.   Buswell,  et  a^.  (1966) 
reported  that  20  percent  of  the  Ph.D*  holders  were  categorized  as 
productive,  while  7.8  percent  of  the  Ed.D«  holders  were  in  the  productive 
category.^*  Another  counter  example  Is  provided  by  Persell  (1971)  whose 
findings  suggest  differences  by  degree  type,  although  the  relationship 
did  not  apply  when  the  holder  of  the  Ed.D.  had  selected  career  research 
experiences  and  was  oriented  to  research  (as  Indicated  by  his  stated 
preference  of  research  over  teaching  and  administration). 


authors  reached  no  conclusions  relating  to  this  finding  since 
degree  type  was  used  as  a  categorization  variable,  and  no  significance 
tests  were  employed  to  determine  if  these  differences  In  productivity  were 
significant.   The  percentages  reported  above  were  merely  presented  In 
tabular  form. 
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The  evidence  on  this  issue  is  equivocal  and  requires  further  study. 
Such  study  might  well  investigate  whether  the  training  received  by  holders 
of  thie  two  types  of  degree  differs  (i.e.,  do  Ph.D.  candidates  systematically 
receive  more  training  in  research  than  do  Ed.D.  candidates)  or  whether 
students  select  for  themselves  the  degree  type  which  fits  more  closely  their 
prospective  professional  role- 

Professional  Experience 

The  interaction  between  graduate  training  variables  and  variables 

related  to  professional  experiences  deserves  attention  at  this  point.  Two 

studies  have  been  concerned  with  this  interaction.    Buswell,  et  al .  (1966) 

found  a  negative  relationship  between  teaching  experience  and  research 

production.    Persell  (1971)  defined  a  variable^  "practice-oriented  social iza- 

tiory'  as  the  graduate  training  or  career  experience  which  would  prepare  a 

person  more  for  the  role  of  educational  practitioner  than  for  the  role  cf 

researcher.       She  found  a  negative  relationship  between  this  variable  and 

15 

her  definition  of  research  quality. 

Worthen  (1969)  analyzed  the  career  patterns  of  existing  researchers 
(based  on  data  from  Bargar,  et  al . ,  1965).   He  found  that  these  researchers 
had  commonly  entered  research  positions  or  begun  research  activities  follow- 
ing occupations  in  a  variety  of  roles  in  professional  education. 

The  set  of  results  in  this  section  again  illustrates  a  conclusion 
alreacfy  posed-  contribution  to  research  is  associated  with  certain  personal 

1 5  " 
Sieber  and  Lazarsfeld  (1966)  came  to  a  conclusion  which  on  the 

surface  appears  related.    This  is  the  conclusion  that  the  requirement  of 

professional  experience  is  negatively  related  to  the  prodjfcti vity  of 

researchers  in  an  institution.   This  latter  variable,  however,  is  defined 

as  institutional  productivity  in  terms  of  numbers  of  persons  who  go  on 

to  be  researchers  and  does  not  relate  to  subsequent  productivity  of  the 

individual  researcher. 
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or  professional  characteristics  and  those  persons  who  are  doing  educational 
research  possess  those  characteristics  more  often  than  educationist's  who 
do  not  participate  In  research  processes.    Information  relating  to  these 
characteristics  has  obvious  Import  for  recruitment  Into  research  training 
programs  or  careers  • 

Employment,    As  of  1964,  the  majority  of  persons  who  Indicated  they 
were  doing  research  In  education  were  not  employed  primarily  as  researchers 
(Bargar,  et  al . ,  1965).      Over  50  percent  of  the  respondents  were  employed 
In  edpjcatlon.  with  psychology  a  close  second  and  sociology  a  distant  third. 
Among  subflelds  of  education,  educational  administration  was  the  largest 
single  category  of  employment.    Fewer  than  20  percent  ot^ll  persons  des- 
cribed In  this  study  as  educational  researchers  were  In  subflelds  of  educa- 
tion which  are  primarily  concerned  with  research.    Similarly,  Puffer  (1967) 
found  that  deans  of  education  Indicated  that  they  Intended  to  hire  only 
limited  numbers  of  faculty  members  with  educational  research  assigned  as 
their  principal  function.   Worthen  and  Roaden  (1969)  reported  that  only 
23  percent  of  the  AERA  members  they  surveyed  held  a  position  which  was  prim- 
arily a  research  position.  This  clearly  points  to  educational  research  as 
a  part-time  activity,  with  limited  financial  support  devoted  solely  to  such 
pursuits. 

The  majority  of  educational  researchers  in  1964  were  employed  In 
colleges  and  universities  (Bargar,  et  al.,  1965).   Within  this  setting, 
only  7.8  percent  were  employed  in  units  Identified  as  being  primarily 
research  units.    It  was  found  that  75.5  percent  of  the  researchers  were 
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employed  in  university  and  college-  units  in  the  field  of  education.  Bargar 
and  his  colleagues  concluded  from  this  and  other  findings  that  almost  a 
quarter  of  those  persons  who  identify  themselves  as  part  of  the  educational 
research  community  have  their  training  and  subsequent  professional  identi- 
fication outside  the  field  of  education. 

The  rather  large  proportion  of  Bargar's  group  of  educational  re- 
searchers  who  are  employed  in  departments  of  education  makes  important 
the  examination  of  these  departments  for  their  research  capabilities. 
Puffer  (1967)  studied  all  institutions  offering  a  bachelor's  degree  in 
education He  found  that  about  22  percent  of  the  faculty  members  of 
education  departments,  schools,  or  colleges   qualify  as  researchers 
according  to  their  dean's  rating  of  their  academic  preparation  in  research 
methodology  and  involvement  in  research  activities.   Membership  in  research 
associations  is  held  by  only  about  17  percent  of  all  education  faculty 
members.    Based  on  these  results  it  is  possible  to  pose  an  hypothesis.  By 
virtue  of  their  background,  training  and  commitment  to  the  field,  those 
persons  employed  in  departments  of  education  should  be  the  primary  pool  of 
persons  performing  independent  research  in  the  field  of  education.  The 
majority  of  these  persons,  however,  spend  small  proportions  of  their  time 
in  research  activities,  and  otherwise  give  indications  of  little  direct 
commitment  to  research  (e.g.,  a  majority  fail  to  meet  the  criteria  which 
Puffer  used  to  qualify  persons  as  researchers).       Most  education  professors 
are   probably      more  committed  to,  and  absorbed  by,  activities  which 
contribute  to  the  improvement  of  educational  practice  —  such  as  teacher 

^^The  findings  of  this  stucly  are  probably  biased  toward  the  large 
institutions,  with  small  institutions  under-represented. 
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training.    Fortunately,  significant  numbers  of  persons  outside  the  disci- 
pline of  education  are  interested  in  contributing  to  educational  research. 

Further  stu<ly  has  been  made  of  the  employment  setting  of  educational 
researchers.   Buswell,  et  al . ,  (1966)  concluded  that  in  view  of  the  pro- 
portion of  researchers  in  academic  settings,  the  major  universities  offer 
the  best  opportunity  for  post- doctoral  research.   They  also  noted  the  small 
number  of  researchers  holding  positions  in  public  schools  at  the  time  their 
stu<ly  was  conducted.    In  addition,  it  was  concluded  that  ".  .  .  while  there 
are  marked  differences  in  the  cases  of  a  few  outstanding  private  institu- 
tions, for  the  country  as  a  whole,  the  amount  of  educational  research  done 
in  public  institutions  as  compared  to  private  institutions  is  quite  im- 
pressive"  (Buswell,  et  al . ,  1966,  p.  28). 

Persell  (1971)  found  that  persons  eniployed  in  either  universities 
or  specialized  agencies  were  more  likely  to  do  research  which  contributes 
to  theory,  practice,  or  research  methodology  than  those  in  other  settings 
such  as  research  units  in  public  schools  or  state  departments  of  education. 
This  could  be  accounted  for  by  the  relatively  greater  preference  for  re- 
search activities  over  other  professional  activities  among  those  persons 
who  work  in  universities  and  research  agencies.   She  also  found  that  pres- 
tige of  the  department  in  which  the  research  is  conducted  relates  differ- 
ently to  type  of  research  contribution  in  various  departments.    For  example, 
while  greater  departmental  prestige  is  related  to  better  contributions 
to  theory  and  research  methods  in  behavioral  science  departments,  in  educa- 
tion departments  greater  prestige  is  related  to  greater  contributions  to 
educational  practice.    This  point  again  emphasizes  the  hypothesized  contrast 
between  orientation  to  educational  practice  and  to  educational  research. 
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Professional  Work  of  Researchers 

The  professional  work  of  researchers  has  been  investigated  in  the 
following  related  areas:    their  research  involvement  (time  spent  in 
research),  their  research  productivity,  the  quality  of  their  research, 
their  identification  with  professional  specializations,  and  their  affilia- 
tion with  professional  associations. 

Research  Involvement.    When  studying  the  amount  of  time  devoted  to 
various  professional  activities  by  self-nominated  members  of  the  educa- 
tional research  community,  Bargar,  et  al.  (1965)  found  that  most  members 
of  the  sample  identified  had  spent  some  time  in  research.    However,  much 
less  time  was  devoted  to  research  than  to  administration,  teaching,  or 
other  activities.   Half  of  the  sample  spent  20  percent  or  less  of  their 
time  in  research.   Of  the  AERA  members  surveyed  by  Worthen  and  Roaden 
(1968),  the  mean  percentage  of  time  spent  in  research  was  18  percent,  and 
the  median  was  10  percent.    Only  37  percent  of  the  respondents 
spent  as  much  as  one  day  par  week  in  research.   About  10  percent  were 
found  to  engage  in  research  administration  and  about  20  percent  were  found 
to  be  teaching  courses  in  research  methods.   Buswell,  et  al .  (1966)  also 
reported  that  a  small  percentage  of  time  was  spent  in  research  by  their 
sample  of  education  doctorates.   They  concluded  that  "In  view  of  the  limited 
time  devoted  to  research,  it  is  not  surprising  that  the  total  output  of 
research  publications  is  so  small"    (Buswell,  et  al.,  1966,  p.  35). 

The  nature  and  demands  of  the  positions  held  by  professionals  in 
education  and  related  areas  probably  imposes  a  restriction  on  the  number 
of  hours  that  potentially  could  be  devoted  to  research.     Clark  (1957) 
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found  this  to  be  true  for  the  psychologists  whom  he  studied.  Wilder  (1966)  i 
found  that  among  reading  experts  the  bulk  of  time  was  spent  in  teaching, 
preparing  classes,  and  counseling.   The  remaining  activities,  including 
research,  tended  to  be  done  on  a  part-time  basis. 

While  the  results  of  other  studies  point  to  a  small  degree  of 
research  involvement  among  educational  researchers,  Persell  (1971)  found 
that  72  percent  of  the  group  she  studied  were  spending  more  than  20  per- 
cent of  their  time  on  research.    Forty-eight  percent  of  her  sample  indi- 
cated that  research  had  been  their  primary  activity  for  some  six  month 
period,  although  only  nine  percent  were  full-time  researchers.    In  terms 
of  their  commitment  to  research,  Persell  found  that  those  who  ranked 
research  higher  than  other  professional  activities  were  also  those  who 
tended  to  produce  research  of  high  quality. 

Population  differences  doubtlessly  account  at  least  partially  for 
this  large  discrepancy  between  the  research  involvement  of  Persell's  sample  and 
that  of  the  other  samples  mentioned  previously.    Persell  began  by  identify- 
ing her  sample  from  those  who  had  published  an  empirical  study,  people  who 
by  definition  were  productive  of  research,  and  therefore  more  likely  to 
Spend  time  in  research  activities.   With  this  counter-example  thus  qualified, 
the  conclusions  seem  to  point  clearly  to  the  fact  that  most  persons  who 
might  be  contributing  to  educational  research  tend  not  to  spend  much  of  their 
professional  time  in  research  activities.    Full-time  educational  research 
positions  appear  to  be  a  rare  phenomenon. 

Research  Productivity.    Buswell,  et  al.  (1966)  used  two  or  more  publi- 
cations in  ten  years  as  the  criterion  for  defining  "productive  researchers." 
According  to  this  criterion,  20  percent  of  the  Ph.D.  holders  and  7.8  percent 
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of  the  Ed.D.  holders  qualified  as  productive.   Wilder  (1966)  found  that 
51  percent  of  the  reading  experts  had  published  at  least  one  article 
since  obtaining  the  doctorate.   Worthen  and  Roaden  (1968)  reported  that 
forty  percent  of  the  AERA  respondents  had  neither  -.-^ceived  a  research 
grant  nor  published  any  research  articles  or  reports  since  receiving 
their  highest  degree.    An  additional  thirty-three  percent  had  averaged 
less  than  one  research  product  (grants  or  publications)  per  year. 
Twelve  percent  averaged  two  or  more  research  products  per  year.  Of 
Persell's  (1971)  group,  15  percent  had  published  only  one  report,  although 
thirty-two  percent  had  published  ten  or  more  papers.    (Again,  the  nature 
of  Persell's  group  must  be  remembered,  since  they  were  chosen  for  produc- 
tivity.)   Relationships  have  been  reported  by  Worthen  and  Roaden  (1968) 
between  research  productivity  and  previous  graduate  experience  as  a  research 
assistant,  when  research  was  the  primary  activity  of  the  assistantship. 
This  relationship  will  be  considered  in  further  depth  in  a  later  section  of 
this  chapter.^ ^ 

Research  Quality.    In  her  analysis  of  the  quality  of  educational 
research,  Persell  (1971)  concluded  that  a  considerable  proportion  (43  percent) 
of  the  papers  analyzed  were  rated  below  average  or  incompetent  with  respect 
to  one  or  more  of  the  three  dimensions  of  quality. '° 


'Sieber  and  Lazarsfeld  (1966)  have  been  cited  as  a  source  of  informa- 
tion about  productivity.    The  definition  of  productivity  used  by  those 
authors,  however,  refers  to  the  rate  of  production  of  researchers  by  an 
institution.  Thus,  those  findings  are  not  included  in  the  present  section. 

^^he  reader  is  reminded  that  Persell's  three  dimensions  of  research 
quality  included  (a)  contribution  to  theory,  (b)  contribution  to  practice, 
and  (c)  use  of  (or  contribution  to)  research  methods. 
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In  examining  those  variables  which  related  to  the  contribution  of 
the  research  to  theory,  practice  or  research  methods,  Persell  found  that 
practice -oriented  socialization  (possession  of  an  Ed.D.,  teaching  exper- 
ience, or  professional  education  courses)  was  negatively  related  to 
those  research  contributions.   Graduate  experience  in  prestigious  depart- 
ments was  positively  related  to  quality,  as  were  career  research  experience, 
research  orientation,  and  intellectual  orientation.   The  author  found 
that  productivity  (number  of  products)  was  not  related  to  contribution  to 
educational  practice  or  the  use  of  research  methods,  but  among  behavioral 
scientists,  was  positively  related  to  contribution  to  theory. 

It  would  appear  from  empirical  results  that  large  proportions  of 
educational  researchers  are  neither  heavily  involved  in  nor  highly  pro- 
ductive of  research.   Of  those  who  are  involved,  significant  proportions 
of  the  work  they  produce  do  not  appear  to  be  making  substantial  contribu- 
tions to  knowledge  in  the  field.    It  may  well  be  that  the  nature  of  a 
profession,  as  opposed  to  a  discipline,  may  dictate  against  research 
pursuits  for  many  professional  personnel. 

Professional  Identification.    In  their  stuc^y  of  educational  researchers, 
BarganeLlL.  (1965)  found  a  high  degree  of  consistency  between  the  fields  of 
present  professional  Identification  and  previous  academic  training.  There 
was  also  a  tendency  fbr  individuals  primarily  identified  with  psychology  and 
social  science  fields  to  have  a  secondary  identification  in  the  field  of 
education. 
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Bargar  and  his  colleagues  also  concluded  that  most  respondents  did 
not  identify  themselves  professionally  in  areas  which  are  primarily  research 
oriented.    The  first  choice  of  respondents  for  their  professional  identifi- 
cation was  education  (51.5  percent),  with  psychology  being  the  second  choice 
(39.7  percent)  and  sociology  the  third  choice  (5  percent) Among  the  sub- 
fields  of  professional  identification,  educational  administration  was  first, 
followed  In  order  by  educational  subject  areas  (combined)  anri  educational 
research.  It  was  found  that  fewer  than  20  percent  of  their  subjects  were 
in  areas  which  they  considered  "obviously  research -oriented,"  such  as 
educational  research,  educational  psychology,  and  educational  sociology, 

Professional  Association.    Information  is  available  on  educational 
researchers'  affiliation  with  the  largest  of  the  professional  organizations 
devoted  to  educational  research,  namely  the  American  Educational  Research 
Association  (AERA).    Morthen  and  Roaden  (1969)  conducted  an  analysis  of  the 
proportion  of  AERA  members  among  authors  of  journal  articles  in  eight  educa- 
tional research  journals  during  the  31  month  period  from  January  1966  to 
July  1968.    They  found  that  only  28  percent  of  this  group  of  persons  who 
produced  educational  research  during  this  period  were  affiliated  with  AERA, 

Worthen  and  Roaden  also  examined  the  research  involvement  and  pro- 
ductivity of  persons  who  were  members  of  AERA.   They  concluded  that  "  he 
majority  of  AERA  members  who  do  research  spend  relatively  little  time  on 
it  In  comparison  with  their  other  professional  activities"  (Worthen  and 


>^  ^^It  is  unclear  whether  Bargar,  et  al_.  considered  the  field  of 

psychology  as  a  whole  to  be  research-oriented. 
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Roaden,  1969,  p.  5),  and  research  is  their  primary  role  less  often  than 
are  other  professional  roles.   Also,  AERA  members  were  not  found  to  be 
heavily  Involved  In  research  administration  or  In  the  teaching  of  research 
methodology  courses,  two  roles  which  facilitate  research.    The  authors 
concluded,  however,  that  If  a  researcher  were  defined  as  anyone  who  either 
spends  one  day  or  more  per  week  in  research  or  spends  one  day  or  more  per 
week  administering  research  or  teaching  a  research  methods  course  (a  lib- 
eral criterion),  as  many  as  60  percent  of  AERA  members  would  qualify  as 
researchers.    If  productivity  is  examined,  however,  Worthen  and  Roaden 
concluded  that  a  majority  of  members  are  not  productive  of  research  when 
one  applies  the  criterion  of  one  research  product  (article  or  report)  per 


The  Researchers  and  the  Settings  for  Research  Activities:    A  Discussion^ 
In  the  preceding  sections,  a  summary  of  relevant  research  results 
has  been  used  to  describe  the  characteristics  of  educational  researchers. 
Those  variables  have  been  discussed  which  distinguish  between  members  of 
the  research  community  who  make  research  contributions  to  the  field  and 
those  who  do  not. 

Empirical  evidence  has  shown  that  contributing  researchers  are  more 
likely  than  noncontributors  to  be  young  upon  receipt  of  the  doctorate,  to 
have  studied  continuously  for  the  degree,  to  have  broader  academic  training 
and  less  educational  practice,  to  have  research  experience  in  graduate 

OA 

Discussion  in  this  section  uses  the  research  results  reviewed 
previously  as  a  springboard  for  speculation  about  probable  relationships 
among  the  variables.    Admittedly,  this    represents  extrapolation  beyond 
the  data  on  the  part  of  the  present  authors. 


year. 


307 


school  and  to  be  irvolved  at  least  to  a  small  extent  in  research  careers. 
Conversely,  when  one  looks  at  the  potential  educational  research  community 
as  a  whole  (all  education  doctorates  plus  persons  in  other  fields  who  have 
an  identification  with  or  interest  in  education  as  a  field  of  stucty),  a 
majority  of  these  persons  are  found  to  be  older  upon  receipt  of  degree,  to 
have  studied  part-time  for  the  doctorate,  to  have  been  involved  heavily  as 
an  educational  practitioner  prior  to,  during  and/or  after  their  doctoral 
stuciy,  to  lack  systematic  experience-based  training  in  research  and  to  be 
engaged  in  professional  positions  where  research  is  not  an  important  or 
predominant  activity. 

Several  explanations  might  be  given  for  this  contrast;  between  the 
characteristics  of  contributors  a-*^  those  of  many  persons  who  by  virtue 
of  their  interest  and  affiliation  with  the  discipline  ought  to  form  the 
contributing  body  of  scholars.   Three  plausible  explanations  are  provided 
below:    (1)  there  is  insufficient  training  in  research  and  ^*cdrce  rumbers 
of  role  models  for  educationists  who  might  become  researches::  '  (2)  there 
are  insufficient  rewards  for  educationists  to  learn  about  and  engage  in 
research  activities  as  opposed  to  practitioner  activities;  and  (3)  the 
interests,  needs  and  motives  of  persons  engaged  in  educational  practice 
are  fundamentally  different  from  those  possessed  by  researchers.  All 
three  explanations  are  in  fact  variants  of  a  single  theme:   whatever  it 
takes  to  nurture,  motivate,  and  support  the  researcher,  both  personally 
and  institutionally,  is  largely  missing  from  (or  its  impact  moderated  in) 
the  development  of  persons  in  the  field  of  education. 

It  i£  interesting  to  speculate  about  the  development  of  both  contri- 
butors an??  f^on contributors  to  research  in  education.   The  career  patterns 
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studies  by  Worthen  (1969)  Indicate  that  the  majority  of  Bargar's  self- 
nominated  educational  researchers  have  origins  within  the  discipline  of 
education.   This  would  mean  that  as  undergraduates  these  persons  opted  for 
professional  training  over  liberal  arts  education.   Personal  orientation 
toward  learning  a  specific  "trade"  or  skill  while  In  college  might  be  an 
explanation  of  the  personal  security  of  a  sijre  career  at  the  end  of  study. 
Humanitarian  and  social  service  goals  are  not  to  be  overlooked. 

At  the  time  he  receives  the  bachelor's  degree,  the  educationist  is 
caught  in  the  mainstream  of  teaching  and  educational  practice  in  general. 
As  a  result  he  is  absorbed  in  practical  concerns  and  day-to-day  problems 
which  he  must  solve.   Research  is  unavailable  or  unknown  to  him  and  thus 
not  used  as  part  of  the  problem-solving  process.    Therefore  he  relies  on 
intuition  and  accumulating  experience  as  sources  of  knowledge  to  use  in 
solving  his  very  Important  everyday  problems.   This  orientation  could  be 
contrasted  with  one  in  which  the  most  valued  end  of  undergraduate  training 
is  in  the  Intellectual  activity  of  absorbing  and  contributing  to  a  body 
of  knowledge.   The  element  of  relative  academic  abilities  must  also  be 
considered. 

Given  these  sets  of  conditions,  the  educator  emerges  from  a  world  of 
everyd«^  educational  problems  to  go  to  graduate  school  for  quite  different 
reasons  than  do  scholars  in  nonapplied  areas.    Among  likely  reasons  are 
the  following: 

1.    Graduate  study  represents  a  wa^  to  advance  in  the  profession 
through  graduate  credit  hour  Increments. 
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2.  Graduate  work  is  a  way  to  change  roles  within  the  system  (again, 
personal  motives).    Here  is  where  women  may  well  fall  out  of 
the  path  to  advancement— probably  for  reasons  of  gender  role 
stereotypes  which  are  incompatible  with  administrative  posts  or 

21 

because  of  overt  or  covert  role  sanctions  in  the  school  setting. 

3.  Graduate  study  provides  specialized  training,  e.g.,  as  school 
psychologist,  counselor,  or  reading  specialist,  thus  facilitat- 
ing role  changes  as  indicated  above. 

4.  More  specialized  training  might  help  attain  personal  social 
service  goals  related  to  finding  more  specific  ways  of  serving 
the  needs  of  children. 

Contrast  these  motives  with  those  of  a  potential  researcher  who 
approaches  gra^Juate  school  as  a  continuation  of  a  process  begun  in  under- 
graduate work— immersion  in  and  contribution  to  a  body  of  knowledge.  Again 
the  academic  ability  variable  intrudes  here.    This  contrast  of  motives,  like 
most  stereotypes,  has  its  limitations.   There  is  little  in  the  research, 
however,  to  contradict  it. 

The  orientation  to  training  would  also  be  different  between  the  two 
groups.   While  a  major  part  of  the  graduate  training  in  most  disciplines 
rx)ns1sts  of  equipping  the  student  with  the  research  tools  needed  to  contri- 
bute to  the  field,  the  educationist  (Indeed  the  applied  psychologist  as 
well)  alms  primarily  at  acquiring  specialized  training  that  will  allow  him 
to  assume  a  new  role  or  advance  within  the  institutional  (i.e.,  clinic  or 
school)  setting.    In  this  context,  research  methodology  plays  a  secondary 

role  fbr  those  individuals. 
  ^ 

^^These  conments  are  not  intended  to  perpetuate  gender  role  stereo- 
types.   Rather  they  are  part  of  an  effort  to  understand  the  reasons  for  the 
sex  conposition  of  the  research Tcowmmity  as  well  as  other  professional  groups. 
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Exposure  to  persons  who  are  trained  in  research,  who  are  engaged 
in  research  activities,  or  even  who  value  inquiry  might  have  an  impact  on 
the  trainees.   A  small  proportion  of  the  faculties  of  schools  of  education, 
however,  qualify  as  researchers  (Puffer,  1967)  and  the  university  faculty 
members  are  primarily  absorbed  with  the  day-to-day  problems  of  educational 
practices,  albeit  on  different  levels  than  the  schoolmen  they  teach. 

Furthermore,  graduate  stu4y  for  educationists  is  generally  a  part- 
time,  noncontinuous  process  which  is  interspersed  between  further  career 
experiences  (Buswell,  et  a1  . ,  1966).   These  experiences  provide  further 
interference  between  the  practitioner  and  any  successful  inculcation  of 
scholarly  values  by  graduate  training  programs. 

When  graduate  studies  in  education  are  viewed  collectively,  there 
emerge   from  this  training  process  persons  who  are  not  adequately  trained 
in  research  models  or  methods  and  who  have  values  which  supersede  systematic 
inquiry.   In  their  subsequent  careers,  there  is  probably  little  incentive 
for  them  to  perform  research,  since  the  production  of  new  knowledge,  per  se, 
is  not  as  valued  in  most  university  education  departments  as  many  other 
activities.    Instead,  the  predominant  orientation  in  most  education  facul- 
ties is  to  develop  solutions  to  problems  on  the  basis  of  position  statements 
or  practical  experience.   In  addition,  the  day-to-day  problems  of  training 
teachers,  administrators,  counselors,  etc.,  are  pressing  and  important  and 
tend  to  exclude  consideration  of  research,  simply  because  there  is  no  time 
left  In  busy  schedules  to  perform  it. 

This  is  not  an  indictment  against  educationists  as  opposed  to  aca- 
demic researchers,  nor  is  it  a  plea  for  recruitment  solely  among  liberal 
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arts  graduates.   Rather,  It  1s  a  statement  that  those  who  are  most  likely 
to  be  committed  to  the  field,  and  therefore  form  the  most  probable  recruit- 
ment pools,  are  by  personal  inclination  not  research  oriented;  they  are 
not  oriented  toward  the  use  of  research  as  a  medium  for  increasing  the 
bo<j|y  of  knowledge  in  the  discipline  and  by  so  doing  solving  its  practical 
problems.    This  lack  of  orientation  mf\y  be  caused  by  personal  ind^' stations 
toward  social  service  goals  and  practical  solution  of  ad  hoc  institutional 
problems  awa^y  from  the  intellectual.  The  point  is  that  the  training 
processes  fail  to  counterbalance  these  inclinations  and  provide  an  in- 
quiry orientation. 

Training  Variables 

This  section  of  the  paper  deals  with  research  which  is  focused  speci- 
fically on  the  training  of  educational  research  and  research-related  personnel. 
The  following  areas  will  be  considered;   the  goals  of  training  programs,  the 
structural  arrangements  of  the  programs,  the  substance  and  experiences  of 
the  programs,  and  the  characteristics  of  recent  trainees  produced  by  the 
programs.    Use  of  information  from  the  previous  sections  on  the  variables 
which  characterize  existing  researchers  will  be  brought  to  bear  on  the 
training  variables  dealt  with  here  to  provide  assistance  in  answering  the 
general  question,  "How  should  training  programs  be  conducted  to  go  beyond 
simply  perpetuating  the  existing  personnel  situation  in  the  educational 
research  community?" 

Coals  of  the  Training  Program 

Ho  definitive  list  of  goals  or  objectives  of  existing  RDOE  training 
programs  could  be  found  In  the  literature  reviewed.  Most  of  the  training 
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programs  studied  by  Krathwohl  (1965),  however,  had  the  training  of  research 
methodologists  as  thei r  principal  objective.   Other  studies  (e.g.,  Fleury, 
1968)  indicated  that  the  stated  objectives  for  most  Title  IV  training  pro- 
grams were  much  the  same. 

In  a  telephone  survey  of  Individuals  who  administer  agencies  in 
which  research  and  research-related  activities  are  conducted  (Sanders  and 
Worthen,  1970),  an  extensive  collection  of  functions  and  competencies  re- 
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quired  of  RDDE  personnel  as  viewed  by  project  administrators  was  identified. 
It  was  concluded  that  the  various  skills  required  under  the  different 
functions  (research,  development,  diffusion,  and  evaluation)  are  inter- 
related.  Although  these  are  identifiable  functions,  they  share  some  common 
characteristics  and  should  not  be  thought  of  as  Independent  of  one  another. 
It  was  also  concluded  that  (a)  in  each  of  several  types  of  institutional 
setting  (e.g.,  colleges  and  universities,  school  districts,  state  educa- 
tion departments)  personnel  my  be  engaged  in  several  research-related 
functions  and  (b)  all  such  functions  are  relevant  to  programs  in  any  of 
the  institutional  settings  surveyed.   "In  terms  of  relative  importance, 
it  appears  that  evaluation,  development,  and  research  rank  in  that  order 
but  all  are  high  and  close  together  on  the  scales  used.  Convemely, 
diffusion  is  viewed  as  relatively^  less  inportant  by  the  majority  of 
ifnvervlewees"  (Sanders  and  Worthen,  1970,  p.  35). 

The  question  raised  in  the  Sanders  and  Worthen  study  remains  unanswered. 
If  the  functions  are  Interrelated,  ought  the  training  programs  to  have  as  their 


The  reader  is  directed  to  Technical  Paper  No.  27  or  Techni cal 
Paper  No.  3  for  a  listing  of  the  coinpetencles  identified  as  Important  in 
perrorming~each  function  and  those  identified  as  being  in  short  supply. 
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objective  the  training  of  skills  generalizable  across  the  four  functions? 
If  the  different  functions  call  for  different  kinds  of  skills,  however, 
the  goals  of  the  existing  training  programs  may  not  be  sufficiently  re- 
sponsive to  the  needs  of  the  community.    In  studying  the  prospective 
roles  for  educational  researchers,  Fleury  (1968)  found  that  94  percent  of 
the  ESEA  Title  IV  programs  prepared  trainees  to  be  researchers  (23  per- 
cent of  the  programs  prepared  trainees  for  development  roles  and  6  percent 
for  diffusion  roles).    He  also  found  that  employers  of  RDD  personnel  call 
upon  them  to  perform  each  function  periodically.    Clark  and  Hopkins  (1969) 
projected  that,  although  deinand  for  trainees  in  1974  would  outdistance  tho 
supply  for  research,  dsvelopment,  and  diffusion,  the  greatest  percentage 
increase  in  demand  would  be  for  development  roles  followed  closely  by 
diffusion  roles.^^ 

It  is  also  possible  to  interpret  the  Sanders  and  Uorthen  (1970) 
results  and  those  of  Fleury  (1968)  as  indicative  of  a  blurring  of  the  dis- 
tinctions among  the  inquiry-related  functions.    Further  research  on 
this  topic  has  been  conducted,  which  considers  both 

the  tasks  performed  and  the  competencies  possessed  by  personnel  who  are 
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performing  work  In  Inquiry- related  positions. 

 1  '.  

'  '"'One  must  be  rea<ty  to  accept  the  assumptions  made  by  Clark  and 
HopMns  In  order  to  accept  these  conclusions  »flth  confidence.    The  reader 
Is  referred  to  Technical  Paper  No.  27  for  rationale  for  qualifying  the 
conclusions  reached  in  their  stu(^. 

^^The  reader  Is  directed  to  Chapter  II  above,  which  first  appeared 
as  Technical  Paper  No.  23  In  the  Task  Force  series.  That  paper  was  not 
completed  at  the  time  studies  in  the  present  chapter  were  reviewed  and 
therefore  is  not  Included  here. 
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Structural  Arrangements  of  the  Training  Programs 

The  question  addressed  in  this  section  is  the  following:  what 
institutional  or  structural  character i sties  of  training  programs  have 
been  associated  with  the  production  of  significant  numbers  of 
researchers?  The  literature  covers  these  topics:    institutional  selec- 
tivity,^^  the  existence  in  the  program  of  opportunities  for  interdisciplin- 
ary training  or  research  apprenticeships,  the  existence  of  research  bureaus 
within  the  institution,  and  the  research  climate  of  the  institution. 

Sieber  and  Lazarsfeld  (1966)  provide  data  on  the  ratio  of  doctorates 

who  enter  positions  which  emphasize  research  to  the  total  number  of 

doctorates  produced  by  an  institution.   The  authors  find  that  high  selectivity 

is  a  good  predictor  of  high  production  of  researchers.   They  conclude 

further  that  the  greater  the  selectivity  the  greater  is  the  output  of 

researchers  for  that  institution.   Those  schools  which  included  professional 

experience  and  a  teaching  certificate  as  part  of  their  entrance  requirements 
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tended  to  be  those  which  had  a  relatively  lower  production  of  researchers. 

In  an  attempt  to  study  the  impact  of  other  institutional  arrangements 
on  the  production  of  researchers,  Sieber  and  Lazarsfeld  (1966)  found  that 


These  are  institutional  arrangements  which  must  be  distinguished 
from  the  effect  of  such  arrangements  on  the  individual  trainee.   The  latter 
is  covered  in  a  later  section. 

^^Selectivity  here  refers  to  the  ratio  of  total  number  of  applicants 
to  the  doctoral  program  to  the  nunber  of  applicants  accepted  in  the  program. 
It  should  not  be  confused  with  an  absolute  level  of  trainee  quality  that 
is  acceptable  for  admission  to  a  program,  although  the  two  may  be  related. 
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It  should  be  noted  that  the  conclusions  reached  by  Sieber  and 
Lazarsfeld  (1966)  were  based  on  descriptive  statistics.  The  production  of 
graduates  entering  primary  research  positions  was  compared  by  looking  at 
the  differences  in  percentage  production  between  categories  such  as  high 
and  low  selectivity.   No  inferential  statistics  were  used,  and  causality 
can  not  be  imputed  to  these  results. 
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both  the  existence  of  a  program  designed  specifically  to  train  research 
methodologists  and  various  measures  of  research  climate  were  related  to 
the  production  of  researchers.   Selectivity  was  a  better  predictor  of  pro- 
ductivity, however,  even  when  the  research  climate  was  unfavorable.   The  inter- 
relationshi,  among  these  variables  was  emphasized  by  Millikan  (1967)  who 
concluded  that  the  production  of  researchers  by  an  institution  was  high 
when  it  possessed  a  "...  cluster  of  organizational  characteristics  important 
for  arrangements  for  research  activity  and  training.    Such  characteristics 
include,  among  others,  a  closed  level  of  admission  to  the  graduate  program, 
a  high  proportion  of  the  graduate  faculty  doing  research,  graduate  prepar- 
ation for  research  emphasized,  a  high  level  of  apprenticeship  on  projects, 
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and  a  program  for  training  in  research"  (Millikan,  1967,  p.  13). 

Sieber  and  Lazarsfeld  (1966)  also  found  that  the  following  measures 
of  research  climate  were  related  to  the  production  of  researchers:  the 
quality  of  research  performed  by  the  institution  (as  judged  by  the 
respondents  surveyed),  the  scope  of  research  (number  of  topics  under  study), 
the  research  emphasis  (high  ranking,  by  deans,  of  research  as  a  faculty  respon- 
sibility), the  existence  of  a  research-oriented  bureau,  and  an  inter- 
disciplinary relationship  index  (a  large  number  of  joint  arrangements  for 
research  with  departments  outside  of  education). 

According  to  another  conclusion  from  Sieber  and  Lazarsfeld  (1966), 
the  existence  of  a  research  apprenticeship  program  is  also  associated  with 
a  high  production  of  researchers,  although  courses  in  research  methods 
are  not.    Again  the  interrelationship  of  several  institutional  variables 


^^The  Millikan  (1967)  study  represented  a  re-analysis  of  the  Sicbor 
and  Uzarsfeld  (1966)  data. 
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is  emphasized  by  Sieber  and  Lazarsfeld.   They  concluded  that  apprenticeship 
experiences  were  an  important  characteristic  of  training  programs  and  were 
related  to  the  presence  or  absence  of  an  interdisciplinary  faculty.  "In 
the  absence  of  an  interdisciplinary  faculty,  apprenticeships  are  not 
positively  related  to  the  production  of  researchers  and  vice  versa.. • 
Exposure  to  an  interdisciplinary  faculty  is  negatively  related  to  the 
production  of  researchers  when  the  level  of  apprenticeship  is  low"  (Sieber 
and  Lazarsfeld,  1966,  pp.  307-308).    Millikan  (1967)  found  that  the 
combination  of  an  apprenticeship  program  with  a  high  proportion  of  economic 
resources  devoted  to  research  was  significant  in  an  institution's  production 
of  researchers. 

In  considering  the  various  relationships  within  bureaus  and  research 
institutes,  Millikan  cites  both  the  volume  of  research  activity  conducted 
within  the  organization  and  the  specific  provision  of  research  training  for 
the  students  who  work  there  as  being  important  in  the  institutional 
production  of  researchers.    In  the  same  context,  Sieber  and  Lazarsfeld  (1966, 
p.  328)  conclude  that  "Units  which  are  affiliated  with  cv^rtain  departments 
in  the  school  of  education  are  much  more  likely  to  provide  opportunities  for 
working  on  projects,  and  indeed  tend  to  favor  assistantships  over  disserta- 
tion work  ..."  They  also  found  that  facilitating  units  (facilitation  refers  to 
whether  or  not  the  unit  facilitates  the  research  of  non-staff  members  or 
mainly  operates  its  own  program  of  research)  less  often  provide  opportunities 
to  work  with  projects,  tend  to  favor  dissertation  work  over  assistantships 
and  more  often  have  students  from  outside  the  school  of  education  employed 
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in  the  unit.   It  was  concluded  that  these  facilitating  bureaus,  regardless 
of  departmental  affiliation,  were  more  productive  of  researchers  than  non- 
facilitating  bureaus,  "...  when  the  school  is  judged  as  doing  outstanding 
research,  when  more  bureau  personnel  are  involved  in  research  than  in  field 
services,  and  when  the  bureau  has  relationships  with  departments  and  schools 

2Q 

outside  the  school  of  education"     (Sieber  and  Lazarsfeld,  1966,  p.  335). 

One  further  set  of  findings  having  to  do  with  the  institutional 
characteristics  of  training  programs  has  been  reported.    The  small  group 
of  most  productive  scholars  studied  by  Buswell,  et  §1.  (1966,  p.  112) 
reported  that  they  had  attended  institutions  with  the  following:  a  strong 
research  climate,  graduate  courses  which  emphasize  (among  other  things)  a 
minimum  of  course  requirements  and  a  maximum  of  independent  study,  a  close 
relationship  with  outstanding,  creative  research  professors,  early  immersion 
in  active  research  projects,  formal  and  informal  contact  with  scholars 
from  a  variety  of  disciplines,  propinquity  of  research  staff  ?ind  facilities, 
broad  flexibility  In  program  offerings,  adequate  physical  facilities, 
libraries,  and  consulting  help  on  research  design.    These  outstanding 
researchers  also  reported,  however,  that  insufficient  economic  resources 
devoted  to  research  projects,  little  opportunity  for  coursework  outside  of 
their  departments,  insufficient  financial  support  for  graduate  students^ 
and  insufficient  training  In  research  design  were  also  characteristic  of 
their  programs. 

The  results  which  deal  with  the  institutional  arrangements  for 
training  programs^seem  to  refer  to  commitment  on  tne  part  of  a  graduate 


^^Agaln,  the  reader  Is  alerted  to  the  causality  which  the  authors 
may  be  implying  in  this  passage.  Such  an  implication  extends  beyond  the 
data  presented. 
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institution  to  give  every  opportunity  for  its  trainees  to  learn  research. 
This  commitment  is  expressed  by  the  monetary  and  personnel  resources  devoted 
to  research  activities  and  the  provision  of  research  bureaus,  apprenticeship 
experiences,  etc.    When  research  involvement  and  a  strong  research  climate 
are  emphasized  in  these  ways,  the  general  value  orientation  may  be  trans- 
mitted to  the  trainees  so  that  they  will  later  opt  for  positions  which 
involve  research.   The  assumption  is  that  the  trainees,  when  trained  under 
these  arrangements,  will  be  both  trained  for  and  oriented  toward  inquiry- 
related  activities. 

Experiential  Content  of  Training  Programs 

One  area  commonly  considered  to  fall  under  the  general  heading  of 
experiential  research  training  is  the  dissertation.   There  is  little  infor- 
mation in  the  literature  about  the  effect  of  the  dissertation  as  a  training 
vehicle.    There  are  indications,  however,  that  it  does  not  provide  the 
training  in  research  that  it  might.    It  was  surmised  by  Buswell,  et  al^.  (1966) 
from  an  anlysis  of  dissertations  completed  by  education  doctorates,  that 
"There  is  a  vast  area  of  dissertation  activity  which  may  not  merit  the 
designation  of  research.    The  methods  are  fuzzy,  the  hypotheses  non-existing 
or  meaningless. . .and  the  statistics  inajipropriate.   No  clear  models  emerged 
in  the  analysis,  which  might  be  used  to  define  and  delineate  the  spectrums 
of  appropriate  dissertation  designs.   The  'product'  or  'project'  is  clearly 
superseding  the  'contribution  to  knowledge'  objectives  in  many  dissertation 
studies"  (Buswell,  et  al-,  1966,  p.  70). 

It  may  be  true  that  a  fair  amount  of  "research  training"  occurs 
after  a  person  receives  the  doctorate  and  is  working  on  a  job  where  research 
is  required.   This  was  sometimes  true  of  the  most  productive  group  of 
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researchers  studied  by  Buswell,  et  §1.  (1966).   Additional  support  for 
this  contention  comes  from  Per s ell 's  study  (1971)  which  indicated  that 
career  research  (i.e.,  post-doctoral)  experience  is  more  important  than 
graduate  research  experience  in  its  relationship  to  research  quality. 

One  type  of  experiential  '^ontent  —  the  research  apprenticeship 
experience  —  is  discussed  in  a  separate  section  below. 

Research  Apprenticeship 

As  noted  in  a  previous  section,  the  existence  of  a  research 
apprenticeship  program  within  a  graduate  institution  (unlike  the  number  or 
type  of  research  courses  offered)  is  related  to  the  production  of  researchers 
by  that  institution.   The  effect  of  the  research  apprenticeship  on  the 

Of) 

individual  trainee  will  be  considered  in  this  section.**" 

The  value  of  involving  a  graduate  student  in  a  research  project  as 
an  apprentice  has  been  discussed  in  various  position  statements  and  has  been 
the  focus  of  several  research  efforts.    Buswell,  et  ai.  (1966)  found  that 
productive  researchers  had  worked  in  a  research  bureau  or  institute  as  a 
research  assistant  significantly  more  often  than  had  non-researchers.  Those 
who  turned  out  to  be  productive  considered  their  assistantships  to  be  more 
than  just  a  means  of  financial  aid.    Further  study  by  Clark  (1957)  indicated 
that  significantly  more  of  the  "significant  contributors"  in  psychology  had 
held  research  assistantships  than  had  psychologists  in  general. 
Similar  differences  were  also  found  between  the  two  groups  with  respect  to 
the  holding  of  other  kinds  of  graduate  assistantships,  suggesting  that 
selectivity  may  explain  the  relationship  between  assistantship  experiences 


^Some  of  the  material  in  this  section  has  been  taken  from  an  existinq 
synthesis  of  research  on  research  apprenticeships  [Morthen,  B.  R.   A  review 
and  synthesis  of  research  on  research  assistantships.    Boulder,  Colorado: 
Laboratory  of  Educational  Research,^jjBrsity  of  Colorado,  1971.  (mimeo)] 
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and  later  productivity.    Students  of  relatively  greater  ability  more  often 

might  have  been  awarded  the  assistantships  or  might  have  gone  on  to  be 

productive  regardless  of  the  apprenticeship  training.   A  counter  example 

to  this  explanation  was  given  by  Millikan  (1967)  who  re-analyzed  the 

Buswell,  et  a^.  (1966)  data  by  dividing  the  respondent  group  into  three 

categories:   those  who  had  held  assistantships  or  fellowships  with  a  research 

component;  those  who  had  held  assistantships  or  fellowships  without  a 

research  component;  and  those  who  had  held  neither  an  assistantship  nor 

fellowship.   The  research  group   differed  from  the  others  by  higher  ratings 

on  four  criteria:    publication  of  a  research  study  which  related  to  their 

dissertation  topic;  participation  in  research  projects;  percentage  of  time 

spent  in  research;  and  preference  for  research  activities.   These  studies 

pointed  to  the  value  of  the  research  assistantship  as  a  training  vehicle. 

In  a  re-analysis  of  the  Bargar  (1965)  data,  Worthen  (1969)  found  in 

his  sample  an  over-all  inverse  relationship  between  research  assistantship 

experience  and  later  research  involvement.  The  author  presented  several 

alternative  hypotheses  which  might  explain  the  discrepant  results  and 

tested  these  in  a  subsequent  study  (Worthen  and  Roaden,  1968).    In  that 

study,  two  general  kinds  of  graduate  research  assistantships  were  differen- 

31 

tiated.    It  was  found  that  experience  as  a  "genuine  research  assistant" 

was  related  to  subsequent  research  productivity  and  research  involvement. 

32 

Experience  in  "ersatz  research  assistantships"     was  found  to  be  unrelated  to 

^^A  "genuine  research  assistant"  was  one  who  served  as  an  apprentice 
to  a  researcher,  worked  in  a  research  bureau,  or  engaged  in  an  internship 
capacity  where  research  was  the  primary  activity. 

^^This  was  a  research  assistantship  which  did  not  involve  research 
as  a  primary  activity. 
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subsequent  research  productivity  or  Involvement.   Those  persons  who  had  held 
an  ersatz  research  assistantship  were  found  to  be  no  more  involved  in  or  produc- 
tive of  research  than  those  who  had  never  held  any  kind  of  assistantship, 

A  basic  conclusion  of  the  study  was  that  "Research  assistantship 
experience  is  a  potentially  valuable  vehicle  for  training  educational 
researchers  who  will  continue  in  research.   Whether  or  not  a  person  actually 
engages  in  research  as  a  major  activity  during  his  assistantship  is  a 
critical  factor  in  determining  whether  the  assistantship  experience  will  be 
positively  related  or  unrelated  to  subsequent  research  involvement"  (Worthen 
and  Roaden,  1968,  p.  186). 

The  significance  of  this  study  was  to  point  out  that  undifferentiated, 
unqualified  research  assistantshlps  are  not  In  themselves  enough  to  insure 
career  productivity.   Other  studies  place  further  qualifications  on  the 
assistantship  experience. 

In  still  a  later  study,  Worthen  and  Roaden  (1971)  studied  the 
specific  training  experiences  of  those  who  had  held  genuine  and  ersatz  research 
assistantshlps  and  found  that  even  among  the  genuine  assistants  there  were 
large  differences  in  the  subsequent  research  productivity  and  involvement 
of  the  subjects.    Among  those  who  had  held  a  genuine  assistantship,  the 
productive  persons  significantly  more  often  than  the  non-productive  group 
had  more  experience  In  the  conceptual  steps  of  the  research  process,  in 
the  use  of  the  computer  for  data  analysis  and  In  the  use  of  statistical 
techniques.   The  unproductive  group,  on  the  other  hand,  was  composed  of 
persons  who  were  more  likely  to  have  engaged  in  clerical  activities  and  to 
have  held  the  assistantship  on  one  specific  project  outside  of  a  research 
bureau  or  laboratory.    Based  on  the  differences  found  among  the  productive 
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and  non-productive  holders  of  genuine  assistantships,  the  authors  made 

the  following  recomnendations  to  research  trainers:    (1)  involve  the 

assistant  in  the  full  range  of  research  tasks.  (2)  help  him  learn  to  apply 

correctly  many  statistical  techniques.  (3)  teach  him  to  use  the  computer, 

(4)  see  that  the  environment  is  conducive  to  research,  (5)  teach  him 

rosearch  techniques.  (6)  make  sure  the  research  to  which  the  assistant  is 

assigned  is  of  high  quality.  (7)  provide  adequate  supervision  and  orienta- 

33 

tion,  and  (8)  manifest  faith  in  the  assistant's  competence. 

Further  qualifications  on  the  assumption  that  the  apprenticeship 
experience  in  anJof  itself  is  related  to  productivity  are  shown  in  other 
studies.    In  an  earlier  section,  the  Sieber  and  Lazarsfeld  (1966)  study 
was  discussed  with  respect  to  the  existence  within  training  programs  of 
apprenticeship  experiences.   The  authors  of  that  study  concluded  that  "... 
apprenticeship  alone  does  not  automatically  guarantee  that  students  are 
sufficiently  prepared  or  motivated  to  pursue  research  careers.    Both  talent 
and  institutional  nurturance  are  necessary  to  reap  the  benefits  of  appren- 
ticeship"  (Sieber  and  Lazarsfeld.  1966.  pp.  302-303).    Still  another  qualifi- 
cation was  identified  by  Persell  (1971)  who  found  that  graduate  research 
experience  was  related  to  research  quality,  but  only  if  that  experience  took 
place  in  prestigious  departments. 

The  res ul las  of  the  latter  studies  point  out  the  danger  of  assuming 
that  research  apprenticeships  are  inherently  valuable.    "A  more  tenable 
assumption  is  that  research  apprenticeship  experience  is  not  inherently 


33rhe  authors  did  not  always  take  note  of  or  adequately  explain  the 
interactions  found  in  the  data.  These  interactions  limit  general izability 
and  impose  a  qualification  on  these  conclusions. 
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valuable  —  rather,  what  a  person  does  on  his  apprenticeship  is  more 
directly  related  to  what  he  does  later  in  his  career  than  is  the  mere 
holding  of  the  position"  (Worthen,  1971,  p.  46). 

Substantive  Content  of  the  Training  Programs 

Information  on  the  content  of  existing  college  and  university  based 
training  programs  is  both  out  of  date  and  limited  in  terms  of  providing 
information  on  those  features  of  the  programs  which  are  most  valuable. 
One  study  (Krathwohl,  1965),  although  dated,  provides  information  on  the 
content  of  the  research  training  programs  in  the  104  institutions  offeinng  the 
Ph.D.  or  Ed.D.  in  1960.   Among, Krathwohl' s  specific  findings  are  the  following: 
91  percent  of  the  institutions  had  an  introductory  course  in  research 
methods,  but  only  13  percent  had  two  or  more  research  methods  courses; 
46  percent  had  an  experimental  design  course;  96  percent  had  a  statistics 
course  (the  mean  number  of  statistics  courses  offered  was  2.2).  There 
was  an  average  of  three  measurement  courses  available,  although  only  a 
third  of  the  institutions  required  one  of  these  courses  for  graduate  degrees. 
Research  experience  beyond  the  thesis  as  a  formal  requirement  appeared  to 
be  rare.    Thirty-eight  institutions  had  programs  (either  operational  or 
submitted  for  institutional  approval)  to  train  research  methodologists. 
Most  of  the  programs  for  methodologists  required  "...  some  kind  of  an 
academic  home  besides  methodology,  the  most  frequent  one  being  educational 
psychology.    Quite  a  few  encourage  work  in  the  basic  social  sciences  as 
wen..."  (Krathwohl,  1965,  pr:^3). 

Title  IV  Programs  in  Research  Training 

The  findings  of  the  Krathwohl  study  are  a  representation  of  research 
training  programs  before  the  impact  of  Title  IV  of  the  ESEA  of  1965  was  felt. 

914  ' 
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One  section  of  this  title  authorized  substantial  financial  support  of 
training  for  excellence  in  educational  research  through  the  "...  development 
and  strengthening  of  research  training  staffs;  to  expand  the  capacity 
and  curricular  capability  for  training;  to  help  in  the  development  of  spe- 
cialized training  programs,  and  through  trainee  stipend  awards,  to  enable 
"'^tk^reater  number  of  persons  to  pursue  careers  in  educational  research" 
(U.i.O.E.,  1965,  p.  3).   Since  its  inception,  the  Title  IV-sponsored  pro- 
grams represented  the  only  major  concentrated  effort  in  this  area.  These 
programs  became  the  primary  vehicle  for  educational  research  training.  The 
literature  offers  descriptive  information  on  the     training  programs  and 
the  trainees  graduating  from  them.   At  a  later  point,  by  relating  the  find- 
ings already  discussed  in  this  chapter  on  the  characteristics  of  the  trainees 
and  the  programs,  some  hypotheses  are  raised  concerning  the  potential  impact 
of  the  Title  IV  program  on  educational  researchers. 

Characteristics  of  the  Title  IV  trainees  have  been  investigated  in 
two  studies.   Sieber  (1968)  and  Hopkins,  et  al.  (1970)  studied  the  1966-67 
and  1969-70  trainee  groups  respectively.    They  observed  that  Title  IV  trainees 
received  their  doctorates  at  approximately  age  31 ,  seven  years  younger  than 
the  1964  group  Buswell.  et  al.    (1966)  studied.    Considering  the  fact  that 
the  average  age  at  the  receipt  of  the  doctorate  had  been  stable  in  the  decade 
prior  to  1964,  it  appears  that  Title  IV  programs  have  significantly  altered 
this  characteristic. 

Considering  further  the  relationship  found  between  early  entry  into 
the  field  and  productivity  of  research  (Buswell,  et  al . ,  1966)  and  between 
relative  youth  of  the  researcher  and  his  research  product  quality  (Persell, 
1971),  it  appears  that  the  programs  potentially  have  made  an  impact  in  an 
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area  which  will  pay  dividends  to  the  field.    Hopkins,  et  al.  (1970)  con- 
cluded that  the  emphasis  upon  younger  trainees  produced  an  increase  in  the 
proportion  of  students  recruited  directly  from  their  previous  degree  program. 
The  proportion  of  trainees  who  entered  the  doctoral  program  immediately 
after  completing  the  bachelor's  degree  changed  from  37  percent  for  the 
« 1966-67  group  to  54  percent  for  the  1969-70  group. 

For  those  trainees  who  were  employed  between  receiving  the  bachelor's 
degree  and  entering  the  graduate  program,  Sieber  (1968)  found  that  66  percent 
had  worked  in  an  educational  setting  which  involved  no  research  activities. 
Two-thirds  of  the  sample  had  a  previous  degree  in  some  area  of  professional 
education.    In  addition,  Sieber  found  that  trainees  were  more  often  committed 
to  professional  education  than  were  researchers  at  large.   These  findings  led 
the  author  to  conclude  that  the  trainees,  as  constituted  at  the  time,  would 
tend  to  Increase  the  concentration  of  professional  education  background  and 
orientation  among  educational  researchers.    Sieber  compared  this  conclusion 
with  that  of  Buswell,  et  al .  (1966)  that  the  most  productive  researchers  in 
the  field  came  more  often  from  outside  professional  education.    The  further 
conclusion  from  Buswell  and  his  colleagues  that  teaching  experience  is  nega- 
tively related  to  productivity  and  the  conclusion  from  Persell  (1971)  that 
practice-oriented  socialization  (graduate  training  or  experience  which  is 
related  more  to  educational  practice  than  research)  is  negatively  related  to 
research  product  qual1i|y,  are  also  relevant. 

Hopkins,  et  al .  (1970)  stated  that  far  fewer  of  the  1969-70  trainees 
had  work  experience  of  any  kind  prior  to  program  entry  than  did  the  Sieber  group. 
The  1969-70  trainees  who  had  not  come  directly  from  an  undergraduate  degree 
were  more  likely  to  have  been  recruited  from  positions  which  involved  some 
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research  activity.   The  1969-70  group  also  was  found  to  have  a  somewhat 
broader  disciplinary  base  than  did  education  students  in  earlier  years,  espe- 
cially in  academic  work  in  the  behavioral  and  social  sciences.  Three-fourths 
of  the  1969-70  group  of  trainees  had  an  undergraduate  major  outside  educa- 
tion.   One  fourth  of  the  trainees  were  studying  for  doctorates  outside 
education.   There  was  also  a  greater  tendency  on  the  part  of  the  1969-70 
trainees  to  seek  the  Ph.D.  rather  than  the  Ed. D.    (69  percent  compared  to 
54  percent  of  the  1966-67  group  and  33  percent  of  the  Buswell,  et  al..  1966 
sample).. 

Hopkins,  et  al.  (1970)  found  that  the  1969-70  group  of  trainees  was 
quite  talented  academically.    Their  Graduate  Record  Exam  and  Miller  Analogies 
Test  scores  were  on  a  par  with  or  higher  than  the  scores  of  the  majority  of 
students  in  nearly  every  professional  and  substantive  field  for  which  scores 
are  available. 

The  Title  IV  trainee  stipends  allow  the  graduates  to  pursue  their 
training  full-time  and  continuously,  a  factor  which  Buswell,  et  al.  (1966) 
found  related  to  subsequent  productivity. 

Concerning  the  characteristics  of  the  trainee  programs  themselves, 
Sieber  (1968)  concluded  that  the  training  programs  were  located  in  the  uni- 
versities which  promised  the  best  contribution  to  research  training.  Hopkins, 
Worthen  and  Soptick  (1970)  agreed.   They  concluded  that,  for  the  most  part, 
the  training  programs  were  located  at  good  training  institutions,  approximately 
two-thirds  of  the  programs  being  at  Institutions  cited  for  the  quality  of 
their  research.   Of  the  remaining  institutions  not  cited  for  research  quality 
per  se,  many  were  nevertheless  noted  for  overall  institutional  quality. 
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This  information  relates  to  the  prior  conclusion  of  Sieber  and 
Lazarsfeld  (1966)  regarding  the  relationship  of  research  climate  (specifically 
the  quality  of  research  done  at  an  institution)  to  an  institution's  pro- 
duction of  researchers. 

Selectivity  also  was  found  to  relate  to  the  production  of  researchers 
by  an  institution  (Sieber  and  Lazarsfeld,  1966).    Information  from  the  litera- 
ture is  limited  with  respect  to  the  selectivity  of  the  Title  IV  training 
programs,  although  Sieber  (1968)  reports  that  the  two  most  often  used  criteria 
for  admission  to  the  programs  were  interest  in  research  and  interest  in  the 
training  program.    Sieber  (1968)  found  tha^liboirt^alf  of  the  programs  used 
the  6RE  scores  for  screening  applicants.   The  MAT  was  used  by  about  one  third 
of  the  programs.    Fleury  (1968)  reported  that  76  percent  of  the  programs 
required  the  GRE  but  only  39  percent  of  these  had  a  specific  score  required 
for  admission.    Forty-four  percent  required  the  MAT  and  of  these,  only  17 
percent  apparently  used  It  actually  to  determine  admission  into  the  program. 
Fleury  (1968)  also  reported  that  48  percent  of  the  programs  designated  a 
specific  minimum  undergraduate  grade  point  average  (34  percent  specified 
a  3.00  or  "B"  average). 

Concerning  professional  experience,  Fleury  (1968)  found  that  45  per- 
cent of  the  programs  required  or  preferred  professional  or  school -related 
experience  or  teaching  or  administrative  experience,  while  55  percent  did 
not  mention  or  require  such  professional  experience.    Sieber  and  Lazarsfeld 
(1966)  had  previously  been  cited  as  the  source  of  the  following  statements: 
♦  •  the  production  of  researchers  (by  an  Institution)  is  negatively  re- 
lated to  the  requirements  of  professional  experience"  (Sieber  and  Lazarsfeld, 
1966,  p,  276) • 
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Sieber  (1968)  found  that  only  a  minority  of  the  Title  IV  programs 
required  interdisciplinary  training.    Approximately  75  percent  of  the 
programs  were  located  in  schools  of  education.   The  substantive  fields  in 
which  research  training  was  being  provided  tended  to  be  fields  of  profes- 
sional education.    In  only  about  one  half  of  the  cases  where  it  was  possible 
to  do  so  were  students  actually  participating  in  the  work  of  educational 
research  organizations  on  their  campuses.    Directors  of  Title  IV  programs 
predominantly  were  located  in  teaching  departments  rather  than  research 
units  and  were  found  more  often  In  fields  of  professional  education  than 
were  researchers  at  large.    These  findings  relate  to  the  conclusion  (Sieber, 
1968)  that  the  training  programs  were  not  utilizing  the  full  range  of  train- 
ing talent  available  In  the  university  setting  (i.e.,  using  noneducation 
personnel  and  resources).    It  should  be  remembered  that  Sieber. and  Lazarsfeld 
(1966)  found  a  relationship  between  interdisciplinary  training  in  an  institu- 
tion and  the  proportion  of  that  institution's  graduates  who  enter  research 
positions.   This  relationship  also  depended  on  the  provision  of  apprentice- 
ship In  the  training  programs.   Sieber  (1968)  found  that  89  percent  of  the 
programs  required  research  work  of  their  trainees.    Fleury  (1968)  found  that 
92  percent  of  the  programs  required  practicum  or  apprenticeship  with  600 
hours  as  the  time  allotment  suggested  by  USOE.   Since  the  nature  of  this  re- 
search work  was  not  explored.  Its  relationship  to  the  research  apprenticeship 
variables  examined  in  an  earlier  section  cannot  be  told. 

In  terms  of  age,  18  percent  of  the  training  programs  specified  "young" 
candidates  (25-<49  years  old)  while  82  percent  did  not  mention  age  as  a 
requirement. 
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The  content  and  substance  of  the  Title  IV  training  programs  have 
remained  the  areas  of  greatest  ignorance  (Hopkins,  et  a1 . ,  1970).  The 
evaluation  reports  submitted  annually  to  the  Research  Training  Branch  of 
the  U.S.  Office  of  Education  should,  but  do  not,  provide  information  beyond 
course  titles  in  this  area.   Fleury  (1968)  in  his  study  of  Title 
IV  programs  found  that  "The  mean  graduate  level  research  training  program 
requires  two  courses  in  statistics  (up  through  analysis  of  variance  and 
covariance),  one  course  in  research  methods,  one  course  in  experimental 
design,  one  course  in  tests  and  measurement,  and  one  course  in  data  process- 
ing"  (Fleury,  1968,  p.  140).    Forty-nine  percent  of  the  training  programs 
required  a  major  in  a  cognate  discipline  and  22  percent  required  a  minor  or 
research  preparation  in  a  cognate  discipline.    It  should  be  remembered,  how- 
ever, that  Sieber  and  Lazarsfeld  (1966)  found  that  the  specific  research 
courses  required  in  a  program  in  the  past  were  unrelated  to  the  institutional 
output  of  researchers. 

It  appears  that  Title  IV  programs  and  trainee  characteristics  appear 
to  approximate  the  arrangements  which  characterize  productive  programs  and 
productive  researchers.   The  trainees  are  younger,  have  greater  academic 
ability,  broader  academic  training  both  before  and  during  their  graduate  pro- 
grams and  have  less  of  the  interference  of  educational  practice.   Their  pro- 
grams are  characterized  by  full-time  continuous  study  (by  virtue  of  trainee 
stipends)  and  systematic  internship  experience  in  research.   The  substantive 
content  of  the  training  programs  and  the  academic  preparation  of  the  prcgram 
trainers  also  may  make  an  impact  by  providing  effective  role  models  for  the 
trainees. 
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Training  Variables;    A  Discussion 

The  results  of  research  dealing  with  training  program  characteristics, 
institutional  arrangements  for  training  programs,  and  the  Title  IV  graduate 
trainees  and  programs  have  been  discussed  in, this  section.   One  might  specu- 
late upon  the  past  and  present  impact  of  Title  IV  on  research  training,  and 
on  the  potential  future  impact  on  research  training  of  the  demise  of  this 
program.    For  by  one  reason  or  another.  Title  IV  programs  succeeded  in  attract 
ing  to  the  field  of  educational  inquiry  persons  of  higher  academic  ability 
than  the  bulk  of  those  who  form  the  present  pool  of  potential  educational 
researchers,  persons  who  are  younger  and  have  broader  academic  backgrounds 
and  less  practitioner  experience— in  short,  persons  who  have  the  character- 
istics which  prior  research  indicates  contribute  to  increased  knowledge  in 
the  field.   By  inference,  these  are  the  persons  who  are  oriented  more  toward 
contributing  knowledge  to  an  academic  field  than  to  engaging  in  educational 
research . 

That  educational  research  should  attract  such  persons  must  be  a 
function  of  the  rewards  offered  by  the  programs— either  money  or  training 
opportunities  of  quality  which  fit  with  their  scholarly  interests.  An 
assumption  which  might  be  made  is  that  when  the  rewards  for  entering  the 
program  are  substantial,  they  will  attract  a  greater  number  of  interested 
persons,  and  the  most  qualified  will  be  accepted.   But  the  program's  success 
cannot  be  predicted  solely  on  these  characteristics;  research  is  needed 
which  will  follow  up  on  the  groups  studied  by  Sieber  (1966)  and  Hopkins, 
et  al.  (1970)  to  check  on  their  productivity. 

Although  data  do  not  exist  to  support  the  statement  that  educational 
research  training  programs  sponsored  by  Title  IV  are  substantively  better 
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(i.e.,  that  their  content  and  experiences  are  superior  to  non-Title  IV  or 
pre-Title  IV  research  training  programs)  improved  substance  and  content 
were  among  the  goals  of  the  Office  of  Education  in  designing  the  Title  IV 
program. 

As  Federally-funded  training  programs  are  reduced  or  phased  out  of 
existence,  the  question  becomes  whether  the  non-monetary  rewards  of  the 
quality  programs  will  provide  enough  motivation  to  attract  high  ability, 
research-oriented  trainees.   It  is  conceivable  that  the  substance  of  the 
programs,  upgraded  as  the  result  of  Title  IV  monies,  might  continue  in  its 
present  form  after  the  funding  is  discontinued.    In  the  case  of  this  eventual- 
ity, the  level  of  trainee  quality  might  not  be  adversely  affected.   If,  how- 
ever, the  monetary  incentive  to  trainees  is  the  crucial  factor,  then  the 
trainee  community  will  revert  to  type  after  the  funds  are  exhausted. 

If  it  is  found  that  the  funding  is  a  crucial  factor  in  attracting 
persons  alreacily  oriented  to  research,  and  this  funding  is  not  available,  then 
it  my  be  necessary  to  find  ws^s  in  which  persons  who  are  already  involved  in 
educational  pursuits  can  become  oriented  to  and  then  effectively  trained  in 
research  through  existing  institutional  arrangements. 
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Suiiinary 

This  review  of  information  on  the  training  of  educational  researchers 
is  limited  to  the  results  of  research  which  have  passed  tests  of  methodo- 
logical adequacy.   Included  are  studies  on  the  characteristics  of  educational 
researchers  and  their  professional  work  and  studies  involving  training 
variables,  such  as  the  selection  of  trainees  and  trainee  characteristics,  the 
goals  and  nature  of  training  programs,  and  the  institutional  setting  for 
the  training  of  researchers.    It  is  clear  from  the  studies  identified  that 
there  is  an  abundance  of  information  on  some  of  the  topics  listed  above  and 
a  major  deficit  in  others.   Information  is  available  on  which  to  base  des- 
criptions of  the  individuals  who  are  educational  researchers.  Information 
is  also  available  to  describe  the  institutional  settings  in  which  the  major- 
ity of  the  training  effort  is  concentrated.    Finally,  information  is  available 
with  which  to  describe  the  general  nature  of  the  research  process.  Major 
deficits  exist  in  the  areas  of  manpower  needs,  the  specific  nature  of  training, 
and  the  nature  of  training  needs  in  the  research-related  roles  of  development, 
diffusion  and  evaluation.  * 

Studies  are  needed  which  answer  the  following  questions  before  long- 
range  planning  can  be  effected  for  a  system  for  educating  the  research  and 
research'^related  personnel  needed  in  education. 

1.   What  are  the  continuing  manpower  needs  in  the  following  roles 
in  education: 

a.-    Production  of  broadly  general izeable  knowledge  about  the 
process  of  education; 
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The  creation  of  the  products  and  procedures  needed  for 
the  conduct  of  education; 

c.  The  distribution  of  Information  among  the  various  special- 
ties In  the  education  system;  and, 

d.  The  generation  of  evaluative  Information  about  educational 
products,  programs,  and  processes. 

.       2.    What  are  the  specific  concepts  and  skills  that  are  needed  to 
perform  in  each  of  these  roles?    (The  answer  to  this  question 
must  go  beyond  the  boundaries  of  empirical  technique;^  the 
method  of  research  encompasses  more  than  sampling,  measuring, 
and  analyzing  data.   The  same  is  true  for  the  other  three  roles.) 
3.    What  procedures  are  effective  in  assisting  students  to  the  mastery 
of  the  concepts  and  skills  referred  to  in  question  two? 
The  answers  to  these  questions  should  not  be  considered  as  static  items,  once 
determined,  to  be  true  for  all  time.    A  vehicle  is  needed  to  obtain  the  infor- 
mation initially  and  to  continually  update  and  refine  that  store  of  information. 
Without  a  continuing  vehicle,  the  effort  to  answer  the  questions  suffers  from 
a  time  constraint  which  will  make  the  information  produced  of  questionable 
value  within  a  decade. 

The  process  of  preparing  educational  research  and  research-related 
personnel  is  a  complex  effort  which  defies  summary  in  a  short  surrimary  section. 
To  attempt  to  do  so  would  do  injustice  to  the  topic.    What  is  known  with 
some  degree  of  surety  is  presented  in  the  preceding  pages.    What  is  not  known 
is  suggested  in  the  questions  above. 
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APPENDIX  A 

INTERVIEW  RECORD  FORM 

Name  of  Interviewee   

Job  Title  

Employer   

Location  

Date  of  Interview   

Time:  Start  

Finish  

Interviewer  • 

MAJOR  AREAS  OF  ACTIVITY 

1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 

11. 
12. 


335 


APPENDIX  B 
INTERVIEW  DATA  SHEET 


ERIC 
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I ntervi  ewee 
Check  ONE: 


B.3 

APPENDIX  B 
INTERVIEW  DATA  SHEET 


Page 


Q  Activities  During  Past  2  Weeks:  Day 
pi  Major  Area  of  Activity  


TASK 


fflMPETENCY 


2. 
3. 
4. 
5. 
6. 

1. 
2. 
3. 
4. 
5. 
6. 

1. 
2. 
3. 

5. 
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Task 
Number 

T  1 


T  2 


T  3 


T  4 


T  5 


T  6 


T  7 


T  8 


C.3 


Task  Categories 

Reading  the  literature  and  acquiring  up-tc-date  information 
througn  other  means^   This  task  Includes  general  reading 
and  inlbrmal  contacts  (e.g.,  discussions  or  correspondence 
with  colleagues)  which  result  in  increased  knowledge  about 
RDDE  or  another  substantive  area. 

Utilizing  formal  search  procedures  to  acquire  information. 
This  Includes  Identification  of  Information  sources  (both 
human  and  material),  acquiring  information  through  retrieval 
systems  such  as  ERIC,  and  the  preparation  of  bibliographies 
and  abstracts  based  upon  the  information  acquired  through 
these  procedures.    The  distinction  between  this  task  and  the. 
previous  one  is  that  the  previous  one  is  general  "keeping 
up  with  the  field"  while  this  task  is  the  systematic  acquisi- 
tion of  information  on  a  particular  topic  for  a  specific 
purpose. 


Conceptualizing  or  formulating  a  problem  or  hypothesis 
empirical  studies.  This  task  Includes  Identifying  and 
delineating  a  problem  for  study  and  determining  what  si 


for 


ing  a  problem  for  stu4y  and  determining 
questions  will  be  investigated  in  the  stucty. 


specific 


Conducting  philosophical  and  historical  analyses.  Included 
are  such  activities  as  formulating  logical  arguments,  con- 
ducting content  or  semantic  analyses,  historiography,  etc. 
Because  both  philosophical  and  historical  analysis  tended  to 
occur  together  in  the  present  sample,  they  are  included 
together  here  as  one  task. 

Identifying  a  product  or  program  which  needs  to  be  developed. 
This  development  task  includes  assessment  of  needs  and  is 
analogous  to  the  research  task  described  in  Nun<>er  3  above. 

Formulating  a  design  for  a  research  study*   This  task  includes 
not  only  tne  choosing  of  a  specific  experimental  design  such 
as  a  Solomon  IV  group  design  or  time  series  analysis,  but  the 
overall  research  design  for  any  type  of  research  stu^y  (e.g., 
ex  post  facto  desi  gn) .  '  ^ 

Formulating  a  design  or  plan  for  an  evaluation.    This  covers 
all  types  of  evaluations  and  Is  the  evaluation  analog  to 
research  Task  6  above. 

Designing  a  specific  educational  development  activity.  This 
task  Is  generally  sequential  to  Task  5,  i.e.,  follows  the 
identifying  of  a  needed  product  or  program.    It  includes  the 
overall  planning  of  the  development  project  and  identificatit)n 
of  various  activities  which  will  be  required  to  develop  the 
product  or  program. 
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Task 

Nunfeer  Task  Categories 

T  9  Conducting,  managing,  or  monitoring  a  fonnative  evaluation. 

Once  a  formative  evaluation  is  designed,  this  task  includes 
all  aspects  of  conducting  the  evaluation,  with  the  exception 
of  those  specific  evaluation  tasks  which  are  identified 
under  other  task  numbers. 

TIG  Conducting,  managing,  or  monitoring  a  suinnative  evaluation. 

This  task  IS  the  summative  evaluation  analog  to  Task  9  above. 

Til  Writing  a  proposal .   This  task  includes  (a)  proposals  which 

are  written  for  the  purpose  of  seeking  funds  and  (b)  proposals 
which  are  written  for  submission  to  persons  who  must  give 
their  approval  before  a  particular  RDDE  activity  can  be 
conducted. 

T12  Formulating  budgets  and  conducting  cost-analyses. 

T13  Utilizing  a  PERT  or  other  management  planning  system.  This 

task  Includes  both  developing  the  system  le.g.,  constructi ng 
a  PERT  network)  and  utilizing  the  system  (e.g.,  using  a 
PERT  network  in  monitoring  an  activity). 

T14  Engaging  in  executive  planning  and  policy  making.   This  task 

includes  making  major  program  decisions  (e.g.,  what  should  be 
the  overall  program  emphasis  of  an  R  and  D  center)  and  making 
personnel  policy  decisions  (e.g.,  should  RAs  be  expected  to 
work  between  semesters). 

T15  Allocating  human  and  material  resources  to  activities.  This 

task  Includes  (a)  day-to-day  decisions  about  allocating  staff 
to  activities  (e.g.,  deciding  how  many  staff  members  will 
work  on  a  specific  activity)  and  (b)  allocating  and  managing  other 
resources  (e.g.,  deciding  whether  or  not  project  monies 
should  be  used  to  purchase  a  particular  piece  of  equipment). 

T16  Communicating  and  negotiating  with  funding  agencies.  This 

task  includes  all  contacts  with  a  funding  agency,  both  before 
funds  are  obtained  and  during  the  time  in  which  funds  are 
being  spent. 

T17  Preparing  RFPs  and  guidelines  for  preparation  of  proposals. 

This  task  is  conducted  primarily  within  funding  agencies  whi ch 
support  RDDE  activities;  however,  persons  in  a  variety  of 
agencies  are  asked  to  assist  in  preparing  such  RFP's  and 
guidelines. 

T18  Reviewing  and  evaluating  proposals  submitted  for  funding. 
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c.5 


Task 
Nurrber 


Task  Categories 


T19 

T20 
T21 

T22 
T23 

T24 
T25 

T26 

T27 

T28 
T29 


Processing  proposals  for  funding  and  monitoring  funded  proposals. 
This  task  includes  activities  which  occur  subsequent  to  a 
decision  to  fund  a  given  activity.    It  includes  dispensing 
funds  as  well  as  monitoring  the  activities  for  which  the  monies 
were  awarded. 

Orienting,  training,  and  upgrading  project  personnel.  This 
task  includes  all  training  of  project  personnel  (in  contras t 
to  Task  63)  and  includes  both  new  and  veteran  personnel. 

Supervising  professional  personnel  .    This  task  differs  from 
Task  15  in  that  it  does  not  include  decisions  about  what  staff 
resources  will  be  utilized  for  a  given  task  but  instead  covers 
the  supervision,  monitoring,  and  assisting  of  personnel  who 
work  on  a  specific  activity.  , 

Supervising  support  personnel.   This  task  is  analogous  to 

Task  21,  but  includes  personnel  such  as  programmers,  technicians, 

and  A-V  personnel . 

Hiring  and  recruiting  personnel.   This  task  includes  recruiting 
paid  employees  of  an  agency  but  does  not  include  activities 
such  as  obtaining  subjects  for  a  studly  or  participants  in  an 
in-service  education  program. 

Designing  and  selecting  facilities  and  capital  equipment. 
Both  buildings  and  equipment  are  included  here. 

Identifying  and  formulating  educational  objectives.   This  task 
includes  identifying  general  goals  and  objectives  as  well 
as  stating  objectives  In  behavioral  form. 

Specifying  and  sequencing  learning  activities.  This  task 
Includes  specifying  and  sequencing  learning  activities  to 
achieve  specified  instructional  objectives. 

Making  projections,  forecasting,  and  analyzing  trends.  This 
task  Includes  both  use  of  logical  analysis  as  well  as  formal 
statistical  techniques  for  projecting,  forecasting,  and 
analyzing  trends. 

Pes  1  gnl ng  curri cul urn  materi al s .  This  task  includes  designing 
materials  to  (a)  include  learning  activities  specified  in 
Task  26  and  (b)  attain  objectives  stated  in  Task  25. 

Writing  or  revising  printed  curriculum  materials.    This  task 
includes  both  the  initial  writing  done  in  the  development  of 
educational  materials  and  subsequent  modification  of  existing 
materials p    It  does  not  Include  the  writing  of  reports  about 
such  educational  materials  or  their  development  (writing  of 
such  reports  Is  included  in  Task  53). 
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Task 
Nuiriber 


Task  Categories 


T30 

T31 
T32 

T33 
T34 

T35 
T36 
T37 
T38 

T39 

T40 


Developing  non-textual  learning  materials.   Examples  of  such 
materials  are  audio  tapes,  films,  and  manipulative  materials. 
This  task  is  analogous  to  29  above  but  is  focused  upon 
learning  materials  other  than  those  in  written  form. 

Producing  or  superyising  the  production  of  curriculum 
materials  Jn  quantity.   This  task  includes  the  mass  pro- 
duction of  materials  which  have  already  been  developed. 

Repairinp  and  maintaining  equipment  (other  than  computers). 
This  tasic  includes  the  actual  repair  and  maintenance  of 
equipment  and  does  not  include  the  supervision  of  personnel 
doing  such  work.    (Such  supervision  is  included  in  Task  22.) 

Planning  data  collection  procedures.    Planning  how  to  collect 
data  (as  opposed  to  actual  collection  of  data)  is  included 
in  this  task. 

Selecting  or  constructing  and  revising  ability  and/or  achieve 
ment  tested   Selecting  an  available  standardized  Instrument 
is  included  here  along  with  constructing  tests  and  revising 
them  through  item  analysis  and  other  techniques. 


^electing  or  constructing  and  revising  measures  of  affect. 
This  task  is  analogous  to  Task  34  above. 

Selecting  or  constructing  and  revising  questionnaires,  check- 
lists, interview  schedules,  and  observation  systems. 

Scaling,  norming  and  establishing  reliability  and  validity 
of  measuring  instruments. 

Conducting  interviews.   This  task  applies  only  to  the  actual 
conducting  of  interviews  and  does  not  include  construction  of 
the  interview  schedule,  which  would  be  included  in  Task  36 
above. 

Administering  group  tests  and  collectinci  data  b^  use  of 
paper  and  pen6il  instruments!  This  task  includes  distributing 
instruments  to  an  intact  group,  such  as  a  class,  as  well  as 
distributing  questionnaires  by  cnail,  etc. 

Admi ni s te ri ng  i n di yi dual  tests .  This  task  includes  activities 
such  as  administering  the  Stanford-Binet  Intelligence  Scale 
and  other  psychological  instruments  which  are  administered 
on  an  individual  basis. 
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Task 
Number 


Task  Categories 


T41 

T42 
T43 

T44 
T45 

T46 

T47 

T48 
T49 


Using  formal  or  informal  observation  systems  to  code  human 
behavior.   An  example  of  this  task  would  be  the  use  of 
F1  anders'  Interaction  Analysis  System  to  code  the  activities 
in  a  classroom;  the  task  does  not  include  the  actual  con- 
struction of  an  observation  system,  which  would  be  included 
in  Task  36  above. 

Tabulating  and  categorizing  data.  This  task  includes  the 
formulation  of  the  categories  into  which  the  data  will  be 
tabulated,  as  well  as  the  actual  tabulation. 

Reviewing  and  critiquing  extant  educational  programs  and 
products.   This  task  includes  making  judgments  about  books, 
hardware,  curriculum  materials,  etc.,  but  does  not  include 
judging  the  results  of  research  or  research-related  activities, 
which  are  included  in  Task  56. 

Field  testing  of  curriculum  materials.   This  task,  which  is 
restricted  to  that  field  testing  which  is  specifically  a  part 
of  a  development  project,  includes  planning,  selecting  sites 
or  subjects,  and  conducting  field  tests. 

Scoring  tests.  This    task  includes  simple  "clerical"  scoring 
as  well  as  that  which  requires  professional  judgment.  It 
includes  both  hand  scoring  and  any  scoring  work  that  might 
need  to  be  done  preparatory  to  submitting  answer  sheets  to 
computer  personnel  for  cooing  and  computer  scoring.  Actual 
use  of  the  computer  for  scoring  is  included  in  Task  49  below. 

Planning  and/or  selecting  data  analysis  techniques.    This  task 
Includes  only  the  planning  or  selecting  of  the  data  analysis 
techniques;  actually  conducting  the  analysis  is  included  in 
other  tasks  below. 

Conducting  data  analyses  by  non- computerized  methods.   Th i s 
task  includes  compiling  data  (other  than  that  formal  cate- 
gorization work  included  in  Task  42  above)  and  doing  computa- 
tional work  by  hand  or  on  a  desk  calculator. 

Developing  a  computerized  data  bank  and  retrieval  system. 

Using  computer  facilities  and  services.    This  task  includes 
selecting  existing  programs,  coding,  designing  formats,  and 
preparing  data  for  analysis  by  existing  programs.    This  category 
includes  computer  scoring  of  tests. 


c.8 


Task  Categories 

Designing  computer  systems .  inspecting  and  diagnosing  computer 
probfems  and  repairing  computer  eguipment.  Note  the  distinction 
between  designing  computer  systems  in  this  task  and  designing 
computer  programs «  which  is  Included  in  Task  52  below.  This 
task  emphasizes  primarily  work  with  computer  hardware. 

Interpreting,  reviewing,  and  integrating  the  results  of  data 
analysis.    Note  that  this  task  applies  only  to  the  interpreta- 
tion  Of  the  results  of  data  analysis  and  does  not  apply  to 
the  data  analysis  itself. 

Designing  computer  programs  and  programming.   This  task  includes 
only  software      computer  design  work. 

Writing  research  and  research-related  reports,  books  and 

arti cles.   Rewriting  and  revising  is  also  included  in  this  task. 

Preparing  and  delivering  a  lecture  or  oral  presentation  on  a 
research  or  research-related  topic.   This  task  is  analogous 
to  Task  53,  but  is  focused  upon  oral ,  rather  than  written 
communication. 

Conducting  conferences  for  dissemination  purposes.  This 
Includes  a11  dissemination  conferences  except  those  in-service 
education  programs  which  are  included  in  Task  63  below. 

Reviewing  and  evaluating  research  and  research -related  reports. 
This  task  Is  analogous  to  Task  43  above,  except  the  focus  here  is 
on  judging  reports  of  research  and  research-related  activities 
rather  than  educational  programs  or  products. 

Preparing  reports,  educational  materials,  and  other  printed 
materials  for  mass  production.  This  task  covers  layout, 
proofing,  organizing,  copy  editing,  etc.   It  is  distinct 
from  Task  31,  which  deals  with  the  production  of  materials 
after  they  have  been  prepared  for  mass  reproduction. 

Disseminating  information  about  activities  on  a  specific  pro- 
ject or  In  a  specific  agency.  This  task  includes  such  diverse 
activities  as  writing  letters  to  disseminate  information, 
handling  phone  calls  in  which  information  is  sought,  and 
arranging  for  interested  persons  to  visit  school  tryout  centers. 

Selecting,  securing,  and  using  consultants.   This  task  includes 
only  the  process  of  obtaining  assistance  from  consultants;  it 
does  not  Include  an  interviewee's  work  as  a  consultant.  Work 
that  an  interviewee  performs  as  a  consultant  would  go  in  the 
appropriate  task  category  which  describes  the  specific  task 
In  which  he  was  engaged  while  serving  as  a  consultant. 
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Task 
Nunfeer 

T60 

T61 


T62 

T63 

T64 
T65 

T66 
T67 


T68 


if?;; 
B 


T69 


Task  Categories 

Negotiating  with  publishers  and  equipment  manufacturers. 

Planning  dissemination  activities.  This  task  is  analogous 
to  Tasks  6,  7  and  8  ftor  research,  evaluation,  and  develop- 
ment. 

Conducting  on-site  evaluation  visits^   This  task  includes 
site  visits  to  evaluate  institutions  or  educational  programs 
or  products  being  used  at  the  site. 

Conducting  in-service  education  programs >    This  task  includes 
planning,  managing,  and  conducting  an  in-service  education 
program,  as  well  as  actual  teaching  within  that  program. 

Translating  written  materials  from  one  language  to  another. 

Developing  and  modifying  information  storage  and  retrieval 
systems >    This  task  includes  such  activities  as  the  develop- 
ment  of  an  ERIC  clearinghouse  system. 

Selecting  and  indexing  documents  for  inclusion  in  information 
storage  and  retrieval  systems. 

Writing  abstracts  of  materials  selected  for  inclusion  in  an 
inTormation  storage  and  retrieval  system >   This  task  Includes 
the  preparation  of  abstracts  for  input Tnto  a  system.    It  does 
not  include  writing  abstracts  for  one's  own  use,  even  though 
tRe  information  being  abstracted  is  taken  from  an  information 
retrieval  system;  this  latter  activity  is  included  in  Task  2 
above. 

Conducting  a  research  stud|y>   This  task  covers  all  activities 
Involved  in  conducting  a  research  study ^  with  the  exception  of 
activities  specifically  listed  under  other  tasks.  Examples 
would  include  obtaining  subjects  for  a  stu()y,  administering 
the  treatment,  etc. 

Conducting  dissemination  activities.   This  task  includes  mis- 
cellaneous dissemination  tasks  such  as  ordering  equipment  for 
a  demonstration  school,  soliciting  cooperation  from  a  school 
to  serve  as  a  demonstration  center,  or  distributing  printed 
dissemination  materials  through  various  means.  ^  It  specifically 
excl udes  the  following  dissemination  activities:    (a)  preparing 
written  materials  of  the  type  Included  in  Task      above,  (b)  con- 
ducting conferences  as  specified  in  Task  55  above,  (c)  dissemina- 
ting Information  about  project  activities  as  specified  in  Task 
58  above,  or  (d)  conducting  in-service  education  programs  as 
specified  in  Task  63  above. 
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Competency 
Number* 


Competency  Categories 


CI  01 


l^nqwledge  of  inquiry  techiriiqugs  in  othgr  disciplines.  

This  refers  to  a  general  knowledge  of  inquiry  techniques 
in  several  disciplines. 


CI 02  Ability  to  discuss  the  advantages  of  establishing  evaluation 

systems  in  educational  institutions.   This  category  also 
includes  the  ability  to  discuss  any  aspects  of  evaluation 
(e.g.,  diminishing  the  threat  of  evaluation)  with  such 
persons  as  school  administrators  or  teachers. 

CI 03  Ability  to  engage  in  effective  oral  communication  with 

others . 

CI 04  Ability  to  listen  effectively.    This  competency  simply  refers 

to  the  ability  to  be  a  good  listener  and  be  sensitive  to 
what  Is  being  said  In  a  discussion. 

CI 05  Knowledge  of  personnel  and  the  organizational  structures  of 

public  school  systems  and  universuiesi   This  knowledge 
entails  a  general  understanding  of  personnel  and  operations 
in  the  field  of  education  at  all  levels. 

CI 06  Ability  to  work  with  public  school,  university,  or  state 

department  of  education  personnel.   ArT example  of  this  competency 
might  be  in  persuading  teacners  or  administrators  in  a 
public  school  system  to  allow  a  research  project  to  be 
conducted  in  that  school  system. 

CI 07  Ability  to  determine  the  evaluative  questions  which  must 

be  asked  in  evaluations  and  the  information  which  must  be 
gathered  to  answer  these  questions. 


*The  numbering  of  competencies  Is  an  outgrowth  of  the  Initial  listing  of 
competencies  In  categories.   Twelve  "look-alike"  categories  were  established 
and  each  was  assigned  a  range  of  digits  (e.g.,  001-099  for  category  1, 
361-368  for  category  5  ,  801-836  for  category  12).   Entries  in  each  category 
were  numbered  sequentially  within  categories  as  they  were  added,  resulting 
In  a  final  numbering  system  which  has  gaps  from  the  final  entry  in  one 
category  to  the  first  entry  In  the  next  (e.g.,  376,  377,  378,  401,  402).  In 
retrospect,  some  early  entries  might  have  been  better  placed  in  another 
catetory;  however,  since  the  numerals  were  only  for  purposes  of  Identifica- 
tion and  clustering  of  factors  was  entrusted  to  the  factor  analysis,  there 
seemed  little  profit  In  moving  competencies  to  fit  a  priori  logical  analysis, 
prior  to  empirical  analysis. 
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Competency 

Nugnber  Competency  Categories 

CI 08  Ability  to  obtain  and  use  feedback  for  management  purposes 

:n  an  on-going  program  or  project. 

CI 09  Ability  to  help  others  identify  and  state  their  objectives. 

This  sKill  relates  to  the  Important  evaluation  function  of 
helping  persons  responsible  for  educational  activities  or 
institutions  identify  and  articulate  the  objectives  of  the 
activity  or  institution.    This  skill  must  be  applied  as  an 
Initial  step  In  constructing  an  evaluation  plan. 

CI  1 0  Knowledge  of  various  evaluation  models  (e.g.>  the  CIPP 

Model  or  Stake's  modeTT! 

cm  Ability  to  be  objective.    This  is  the  ability  to  avoid 

injecting  one's  own  values  or  biases  into  an  inquiry  activity. 

CI  12  Ability  to  design  or  conduct  interviews  for  the  purpose  of 

collecting  data.    This  competency  is  in  contrast  with 
Competency  617  which  includes  job  Interviews,  which  have  as 
their  primary  purpose  the  hiring  of  personnel. 

CI  13  Ability  to  involve  the  community  In  evaluation  projects  or 

developing  educational  programs.   This  competency  refers 
to  the  ability  to  use  community  input  in  establishing 
priorities,  assessing  how  widely  program  objectives  are 
shared  by  the  community,  etc. 

CI  14  Ability  to  translate  data  analyses  into  recommendations  for 

action. 

CI  15  Ability  to  incorporate  systematic  evaluation  procedures  into 

plans  for  developing  educational  programs. 

CI  16  Ability  to  develop  techniques  for  providing  evaluative  feed- 

back  to  program  or  project  personnel  in  time  to  allow  needed 
modifications  to  be  made  during  the  operation  of  the  program. 

CI  17  Ability  to  Identify  the  decision  irakers  who  need  evaluative 

feedback. 

CI  18  Ability  to  secure  cooperation  from  persons  necessary  to 

conduct  an  inguiry  or  inquiry-related  activity.    An  exampl e 

of  this  competency  would  be  getting  school  admi n i s trators 

to  par^tlcipate  as  subjects  in  a  study  of  administrative  styles. 

CI  19  Knowledge  of  factors  which  increase  or  decrease  credibility 

of  evaluation  reports^ 

C201  Knowledge  of  and  ability  to  plan  data  collection  procedures 

appropriate  to  a*st)ecific  inquiry  activity. 
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Competency 

Number  Competency  Categories 


C202  Knowledge  of  measurement  theory  and  techniques. 

C203  Knowledge  of  criterion-referenced  testing. 

C204  Knowledge  of  general  principles  of  instrument  construction. 

C205  Knowledge  of  questionnaire  construction  techniques  and 

appropriate  uses  for  questionnaires. 

C206  Ability  to  construct  Instruments  to  assess  attitudes  and 

other  affective  variables.   The  ability  to  use  personality 
tests  is  included  in  this  competency. 

C207  Ability  to  select  appropriate  standardized  tests  or  instru- 

ments.  This  ability  would  require  knowledge  of  principles 
of  Instrument  construction  but  would  go  beyond  that  to  also 
require  knowledge  of  sources  of  reliable  information  about 
standardized  instruments. 

C208  Knowledge  of  norming  procedures. 

C209  Knowledge  of  theory  and  techniques  for  assessing  student 

achievement^ 

C210  Knowledge  of  systems  developed  to  categorize  human  behavior 

or  abilltl^   Examples  of  such  systems  would  be  (a)  a 
method  of  task  analysis  such  as  Gagne's  "backward  chaining," 
or  (b)  Bloom's  taxonomy  in  the  cognitive  domain. 

C211  Ability  to  construct  Items  that  measure  what  one  sets  out 

to  measure"    An  alternate  way  of  describing  this  competency 
is  "the  ability  to  achieve  construct  validity"  for  a  given 
Instrument. 

C212  Ability  to  write  unambiguous  Items  In  vocabulary  appropriate 

to  the  specified  audience.   This  competency  requires  a 
knowledge  of  the  vocabulary  level  of  the  prospective 
respondents  to  an  instrument. 

C213  Ability  to  construct  Items  In  such  a  way  that  they  increase 

the  likelihood  of  response.    Included  in  this  competency 
would  be  the  ability  to  construct  items  which  (a)  are  not 
Inflanmatory  in  nature,  (b)  are  not  overly  threatening,  and 
(c)  motivate  the  respondent  to  complete  thciii. 

C214  Ability  to  determine  a  logical  sequence  for  all  items  in  an 

Instrument. 


C215 


Abnity  to  construct  good  multiple  choice  or  other  types  of 
Objective  Items. 


Competency 
Number 


Competency  Categories 


C217  Ability  to  arrange  a  response  format  for  items  which  will 

facilitate  later  coding  and  analysis  of  the  data^ 

C218  Ability  to  conduct  item  analyses,  including  computing 

difficulty  and  discrimination  indices. 

C219  Ability  to  design  a  clear  instruction  manual  for  the 

administration  of  tests. 

C220  Knowledge  of  and  ability  to  use  unobtrusive  measurement 

techniqueTT 

C221  Ability  to  construct  good  rating  scales. 

C222  Ability  to  construct  culture-free  measures. 

C223  Ability  to  arrange  items  in     a  format  which  is  easy  to  read. 

This  competency  includes  determining  the  proper  number  of 
items  per  page,  good  printed  or  typed  formats,  etc. 

C301  Knowledge  of  different  types  of  research  methods.    This  is 

a  very  broad  competency  relating  to  an  understanding  of 
different  types  of  research  methodology,  such  as  empirical 
research,  historiography,  etc. 

C302  Knowledge  of  survey  research  designs  and  techniques. 

C303  Knowledge  of  general  principles  of  research  desicjn. 

C304  Knowledge  of  specific  experimental  and  quasi -experimental 

research  designs.  Knowledge  of  the  Campbell -Stanley  desi gns 
would  be  included  here.*  ^ 

C305  Knowledge  of  factors  which  Jeopardize  internal  and  external 

validity.    Again,  knowledge  of  Campbell  and  Stanley's  1 i s t 
of  threats  to  validity  would  be  included  here. 


♦Campbell,  D.  T.,  &  Stanley,  J.  C.   Experimental  and  quasi -experimental 
designs  for  research  on  teaching.    In  N.  L.  Gage  (Ed.),  Handbook  of  research 
on  teaching.   Chicago:    Rand  McNally,  1963.    (Reprinted  as  Experimental  and 
quasi-experimental  designs  for  research.    Chicago:   Rand  McNally,  1966.) 
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Competency 

Number  Competency  Categories 

C306  Ability  to  operational ize  a  research  or  evaluation  design 

into  specificj)rocedures  for  conducting  the  stuHy^ 

C307  Ability  to  design  studies  to  control  extraneous  variables. 

Included  In  this  competency  would  be  knowledge  of  appro- 
prlate  uses  of  a   control  group   and  other  means  for 
exercising  control  In  a  study  to  arrive  at  unequivocal 
conclusions. 

C308  Ability  to  Identify  variables  which  should  be  included  in 

a  research  or  evaluation  study^ 

C309  Knowledge  of  sampling  theory  and  techniques. 

C310  Knowledge  of  methods  for  planning  or  installing  a  complete 

curriculum  or  a  curriculum  package. 

C311  Knowledge  of  case  study  designs. 

C312  Ability  to  plan  effective  development  procedures. 

C313  Ability  to  identify  the  problem  and  articulate  the  problem 

statement  in  a  research  or  evaluation  study. 

C314  Ability  to  formulate  testable  hypotheses  or  answerable 

questions  in  a  research  or  evaluation  study. 

C361  Knowledjje  of  educational  theories,  practice,  and  terminology. 

This  knowledge  reflects  a  general  acquaintance  with  estab- 
lished  theories,  terms,  and  practices  used  in  the  field  of 
education. 

C362  Knowledge  of  the  objectives  or  learning  sequences  involved 

in  a  given  curriculum  or  set  of  curriculum  materials.  ThTs 
knowledge  is  prerequisite  to  designing,  field  testing,  or 
critiquing  a  given  curriculum  or  curriculum  package. 

C363  Knowledge  of  current  theories  of  learning,  especially  as 

they  relate  to  theories  of  instruction. 

C364  Knowledge  of  specific  socioeconomic  or  ethnic  cultures. 

C365  Knowledge  of  a  foreign  language  or  various  English  dialects. 

f 

C366  Ability  to  seguence  learning  activities  to  facilitate  student 

learning  in  a  curriculum  or  set  of  curriculum  materials. 
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Competency 

Number  Competency  Categories 


C367      1     Knowledge  of  protocol  materials  and  how  they  can  be  used 
and  demonstrated. 


C368 


Ability  to  use  field  testing  technigues  during  preliminary 
tryout  or  implementation  of  new  curriculum  materia!T 

.C369  Ability  to  design  or  select  curriculum  materials  to  match 

the  instructional  "style"  of  the  classroom  for  which  they 
are  intended.  \ 

C370  Knowledge  of  how  to  make  use  of  various  media  and  media 

resource  personnel" 

C371  Ability  to  draw  or  compose  pictures  or  illustrations  for 

curriculum  materials. 

C372  Ability  to  develop  or  organize  a  teacher's  guide, 

C373  Knowledge  of  sociology  and/or  sociological  research  methods. 

C374  Knowledge  of  developmental  psychology  or  the  field  of 

psychology  in  general.   This  competency  requires  either  an 
understanding  of  child  development  (knowledge  useful  to  a 
person  who  develops  curriculum  materials)  or  an  understand- 
ing of  any  aspect  of  psychology  listed  by  interviewees  as 
necessary  knowledge  In  their  work. 

C375  Ability  to  incorporate  individualized  instruction  into  the 

design  of  curriculum  materiaT?! 

C376  Knowledge  of  procedures  and  steps  in  developing  curriculum 

ilS. 


C377 


material 


Ability  to  organize  recently  developed  educational  materials 
to  enable  them  to 'be  distributed  promptly  upon  request. 

C378  Ability  to  use  various  media  (TV,  tapes>  slides,  pamphlets > 

etc.)  to  disseminate  information  on  innovative  educational 
practices  or  products. 

G401  Knowledge  of  relevant  subject  matter.   This  competency  refers 

to  an  understanding  of  a  substantive  field,  including  know- 
ledge of  major  theories,  well-known  authors  in  the  field, 
the  current  trends  in  the  field,  etc. 
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Competency 
Number 


Competency  Categories 


C402 


i 


I 
I 
I 


C403 

C404 
C405 

C406 
C407 
C408 
C409 

C410 
C411 

C412 

C413 

C414 


Knowledge  of  current  educational  Innovations  (e.g..  Indivi- 
dually prescribed  Instruction)  and/or  recent  developments 
In  the  JField  of  education.   This  competency  relates  to  a 
knowledge  of  what  Is  happening  today  in  the  field  of  educa- 
tion —  not  necessarily  an  acquaintance  with  specifics  of 
education,  but  an  awareness  of  major  Innovations  and  major 
Issues  or  occurrences  In  the  field. 

Knowledge  of  classification  systems  of  educational  goals. 
This  competency  Includes  knowledge  about  taxonomies  in  the 
cognitive,  affective  and  psychomotor  domains. 

Ability  to  be  creative. 

Ability  to  use  analytical  or  logical  skills.   These  skills 
would  Involve  such  abilities  as  breaking  things  down  into 
component  parts  or  perceiving  logical  contradictions. 

Ability  to  draw  correct  inferences,  conclusions,  or  general- 
izations. 

Ability  to  recognize  and  use  relationships  among  concepts, 
processes,  or  other  phenomenal  ~ 

Ability  to  formulate  a  rationale  to  support  a  particular 
.  position  or  argument. 

Ability  to  evaluate  or  critique  a  written  or  oral  presentation. 
This  competency  includes  the  ability  to  formulate  constructive 
criticism  when  writing  reviews,  reading  critically,  etc. 

Knowledge  of  mathematical  logic. 

Ability  to  apply  abstract  theory  to  concrete  problems  or 
situations.  [ 

Ability  to  predict  with  accuracy  the  Impact  of  an  Inquiry 
or  inquiry-related  activity. 

Ability  to  conceptualize  or  ''brainstorm''  new  Ideas.  This 
is  the  general  conceptual  skill  of  initiating  ideas  one 
wishes  to  pursue. 

Knowledge  of  systems  analysis  concepts  and  techniques. 
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Competency 
Number 


Competency  Categories 


C415 

C416 
C417 


C418 


C419 
C420 


I  . 


C421 
C422 

C501 

C502 


C503 
C504 


Ability  to  be  open-minded.   This  competency  could  involve 
the  ability  to  break  a  mental  set  or  the  ability  to 
tolerate  ambiguity. 

Ability  to  pinpoint  important  issues  or  themes. 

Ability  to  make  long-range  forecasts  or  predictions.  This 
competency  would  include  both  the  "crystal-ball"  ability  to 
think  about  the  hypothetical  or  the  improbable,  and  know- 
ledge of  formal  methods  of  forecasting,  such  as  the  Delphi 
technique  or  the  Cross  Impact  Matrix. 

Knowledge  of  strategies  of  educational  change.  This 
competency  might  include  an  acquaintance  with  theories  of 
change,  knowledge  of  how  to  use  change  agents  or  other 
factors  to  facilitate  or  induce  change  in  a  particular 
situation. 

Knowledge  of  ways  in  which  educational  policies  are  formulated. 

Knowledge  of  legalities  related  to  inquiry  or  inquiry-related 
projects.   Such  legalities  might  Include  the  rights  of 
subjects  being  Investigated  in  a  research  or  evaluation  study 
(e.g.,  constraints  related  to  Invasion  of  privacy  statutes) 
or  copyright  laws,  relating  to  curriculum  materials  produced 
on  a  development  project. 

Ability  to  state  objectives  in  measuireable  terms. 

Knowledge  of  various  uses  of  the  computer  in  education.  An 
example  of  this  competency  Would  be  knowledge  of  a  system 
such  as  Computer  Assisted  Instruction  (CAI). 

Knowledge  of  steps  Involved  in  the  mass  production  of 
curriculum  materials  (e.g.,  reproduction  and  packaging 
processesTi 

Knowledge  of  printing  constraints  and  specifications.  This 
competency  would  require  knowledge  of  factors  such  as 
various  type  sizes  or  type  faces  and  the  ability  to  communi- 
cate specifications  to  the  printer. 

Ability  to  make  arrangements  with  publishers. 

Ability  to  arrange  purchases  of  needed  materials  to  conduct 
a  project  or  administer  an  organization" 
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Competency 

Number  Competency  Categories 


C505 


C545 


Knowledge  of  photographic  processing  or  the  technical  aspects 
inyolved  in  the  production  of  artwork.   This  would  include 
a  knowledge  of  such  processes  as  silkscreening  or  color 
overlays. 


C506  Ability  to  take  photographs. 

Knowledge  of  design  st 
multinnedia  materials. 


C507  Knowledge  of  design  stages  in  developing  audio-visual  or 


C508  Ability  to  develop  appropriate  product  specifications  for 

new  educational  products  and  ability  to  determine  when 
specifications  are  met  well  enough  to  warrant  dissemination 
of  the  product. 

C509  Knowledge  of  workshops  and  the  flow  of  materials  therein 

(in  regard  to  the  production  of  visual  aids  or  other  educa- 
tional materials).  ~ 

C541  Knowledge  of  instructional  approaches  that  might  be  incor- 

porated in  teaching  or  designing  instructional  materials. 
This  knowledge  mignt  include  an  acquaintance  with  various 
teaching  models,  such  as  discovery  learning  or  the  open 
classroom. 

C542  Ability  to  assess  students'  attitudes  toward  an  educational 

program,  product,  or  practice.   Such  knowledge  is  potentially 
useful  In  designing  or  implementing  curriculum  materials  as 
well  as  in  evaluating  them. 

C543     ^      Knowledge  of  the  role  of  the  teacher,  including  abilities 
which  normally  can  be  expected  of  teachers.    Again,  such 
knowledge  would  be  of  use  to  those  developing,  disseminating, 
or  evaluating  curriculum  materials. 

C544  Ability  to  make  necessary  arrangements  for  conducting  in- 

service  Institutes  or  workshops. 

Ability  to  teach  effectively.   This  competency  refers  to 
the  ability  to  teach  or  demonstrate  educational  products  or 
practices. 

C546  Knowledge  of  parents'  attitudes,  abilities,  and  the  informal 

academic  training  they  give  to  their  children. 

C547  Knowledge  of  methods  of  teacher  training. 
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C548  Ability  to  use  manipulative  classroom  teaching  materials. 

Such  materials  would  Include  laboratory  equipment  that 
students  coul clause  in  learning  elementary  concepts  of 
science,  or  building  blocks   or  an  abacus  that  might  be 
used  to  teach  basic  math  concepts  to  elementary  children. 

C549  Knowledge  of  appropriate  instructional  materials  to  be 

used  with  students  at  certain  age  levels. 

C601  Ability  to  administer  any  type  of  project  or  program.    ( 1 1 

was  necessary  to  Include  such  a  broad  competency  as  this 
because  some  interviewees  could  not  break  down  their  admin- 
istrative skills  into  any  further  specifics.) 

C602  Knowledge  of  organizational  or  management  theory. 

C603  Knowledge  of  inquiry  or  inquiry-related  management.  This 

refers  to  the  specific  knowledge  of  how  to  manage  research 
or  research-related  projects. 

C6p4  Knowledge  of  the  role  of  inquiry  and  inquiry-related  activities 

in  education.    This  Includes  Insight  into  what  is  meant  by 
^  terms  such  as  educational  research,  development,  diffusion 
and  evaluation  and  an  understanding  of  how  they  relate  to 
one  another. 

C605  Ability  to  develop  an  organization  or  organizational  unit. 

This  competency  relates  to  the  broad  knowledge  required  to 
organize  or  set  up  a  unit  such  as  an  R  &  D  center  and  would 
include,  for  example,  a  knowledge  of  staffing  plans. 

C606  Knowledge  of  rational  decision-making  processes.  This 

competency  would  Include  a  knowledge  of  such  aspects  of 
administration  as  how  to  establish  priorities  or  how  to 
weight  alternatives  in  making  decisions. 

C607  Knowledge  of  the  organization  for  which  you  are  working. 

Including  knowledge  of  its  needs,  resources,  methods  of 
.  operation,  etc. 

C608  Ability  to  supervise  personnel .   This  broad  competency  relates 

to  worKing  with  people  who  are  responsible  to  you  and  would 
include  assigning  tasks,  monitoring  the  work  of  individuals, 
motivating  staff,  providing  guidance  and  leadership  for  these 
personnel ,  etc. 
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Competency 

Number  Competency  Categories 


C609 


Knowledge  of  effective  techniques  for  writing  and  submitting 
proposals  to  obtain  funding.    Included  here  would  be  a 
knowledge  of  the  process  one  must  follow  in  submitting 


proposals. 

C610  Ability  to  identify  and  obtain  resources  needed  to  accomplish 

program  objectives.   Such  resources  could  be  financial  or 
could  Involve  necessary  facilities,  personnel,  and  so  forth. 


C611  Knowledge  of  personnel  evaluation  practices.    This  competency 

refers  to  (a)  the  ability  to  judge  a  person's  potential  value 
prior  to  employing  him  and  (b)  the  ability  to  determine  if  an 
individual  is  performing  adequately  in  his  job. 

C612  Ability  to  design  a  physical  plant  or  plan  the  facilities 

needed  for  a  program  or  project^   This  competency  includes 
a  knowledge  of  what  kinds  of  physical  needs  are  required 
to  carry  out  inquiry  or  inquiry-related  activities,  and 
also  involves  the  ability  to  translate  space  needs  into 
physical  plans. 

C613  Ability  to  identify  educational  needs  that  should  be 

addressed  by  educational  systems. 

C614  Ability  to  determine  financial  resoorces  necessary  to  conduct 

a  pirogram  or  project  and  use  accounting  procedures  to  operate 
within  a  program  or  project  budget.  '■ 

C615  Ability  to  plan  and/or  coordinate  activities  for  a  program 

or  project. 

C616  Knowledge  of  aaid  ability  to  use  management  and  planning  systems 

such  as  PERT  (Program  Evaluation  and  Review  TechnTquiyT 
PPBS  (Program  Planning  Budgeting  System),  or  Critical  Path 
Analysis. 

C617  Knowledge  of  effective  techniques  of  recruiting,  interviewing 

and  hiring  personneT? 

C618  Ability  to  "stick  with  a  task"  to  its  conclusion. 

C619  Ability  to  work  with  small  but  Important  details  of  larger 

project  activities^ 

C620  Ability  to  outline  specific  procedures  for  working  through 

a  problem" 
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Competency 

Number  Competency  Categories 

C621  Ability  to  "trouble  shoot"  or  resolve  problems  in  an  ongoing 

project. 

C622  Ability  to  facilitate  staff  work  on  an  ongoing  project. 

C623  Ability  to  influence  others  to  attain  project  or  program 

objectives^  I 

C624  Ability  to  handle  routine  professional  correspondence  and 

responsibilities. 

C625  Ability  to  make  progress  assessments  for  ongoing  activities. 

C626  Ability  to  estimate  realistically  the  time  required  for 

various  inquiry  and  inquiry-related  activities. 

C627  Ability  to  maintain  work  schedules. 

C628  Ability  to  conduct  effective  public  relations. 

C629  Ability  to  work  effectively  with  decision  makers.   This  skill 

would  include  the  ability  to  press  for  an  idea  with  super- 
visors or  those  in  authority,  as  opposed  to  acquiescing 
when  dealing  with  persons  in  authority. 

C630  Knowledge  of  political  considerations.   This  knowledge  refers 

to  an  awareness  of  political  factors  within  institutions, 
local  communities,  regions,  or  the  nation. 

C7dl  Knowledge  of  formal  or  informal  systems  of  recording  observa- 

tions of  behavior.   TFis  competency  would  include  acquaintance 
with  the  purposes  and  operations  of  such  formal  systems  as 
those  designed  by  Flanders  or  Bel  lack. 

C702  Ability  to  use  formal  or  informal  systems  of  recording 

observations  of  behavior.  This  skill  would  relate  either 
to  Implementing  such  a  system  in  an  institution  or  to  the 
actual  coding  of  observations  in  the  classroom. 

C703  Ability  to  develop  an  observation  system  of  coding  behavior 

in  the  classroom.  ~ 

C726  Ability  to  read,  interpret,  and  follow  an  administration 

manual  for  standardized  testis 

C727  Ability  to  present  directions  clearly  in  administering  an 

instrument.  ~' 
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Competency 

Number  Competency  Categories 


C728 


Ability  to  adjust  test  administration  procedures  when 
situational  factors  make  such  adjustment  essential. 
An  example  of  this  competency  would  be  the  ability  to 
adjust  administration  procedures  in  an  individual  testing 
situation  when  one  determines  a  subject  is  not  giving  valid 
or  reliable  responses. 

C729  Ability  to  establish  rapport  with  children  and  obtain  their 

cooperation  In  testing  situations. 

C801  Ability  to  choose  (or  design)  appropriate  statistical 

techniques  for  data  analysis. 

C802  Ability  to  manipulate  data  so  as  to  extract  all  relevant 

information.  '  ~ 

.  C803  Ability  to  determine  the  relevance  of  new  information  or 

educational  materials  for  the  field  of  education. 

C804  General  knowledge  of  statistics.   This  competency  was  coded 

only  when  a  respondent  could  not  delineate  specific  statis- 
tical skills  but  still  mentioned  he  needed  statistical 
knowledge  in  carrying  out  his  work;  it  simply  refers  to  a 
general  understanding  of  the  uses  of  statistics  and  acquain- 
tance with  some  of  the  more  commonly  used  techniques. 

C805  Knowledge  of  descriptive  statistical  techniques. 

C806  Knowledge  of  or  ability  to  calculate  various  kinds  of  test 

scores  or  scoring  procedures.  Examples  of  such  scores  would 
be  grade  equivalent  scores,  stanine  scores,  or  gain  scores. 

C807  Knowledge  of  ANOVA  or  ANCOVA  designs  and  techniques. 

C808  Knowledge  of  correlational  techniques. 

C809  Knowledge  of  multiple  correlational  techniques. 

C810  Knowledge  of  statistical  regression  techniques. 

081 1  Knowledge  of  t-tests  and  critical  ratios. 

C812  -Knowledge  of  statistical  variance  and  standard  deviation. 

C813  Knowledge  of  instrument  reliability  including  types  of 

reliability  coefflclenis^    ~  '.  . 
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Competency 

Number  Competency  Categories 

C814  Knowledge  of  or  ability  to  detennine  instrument  validity, 

including  various  approaches  to  determining  validftyi 

C815  Knowledge  of  factor  analytic  techniques. 

C816  Knowledge  of  distribution  theory  and  distributions  commonly 

used  in  statistics." 

C817  Knowledge  of  non-parametric  statistical  techniques. 

C818  Knowledge  of  methodology  for  inferring  causation  from  non- 

experimental  data. 

C819  Knowledge  of  theoretical  assumptions  underlying  various 

statistical  technigues. 

C820  Ability  to  use  printed  statistical  tables  (e.g.,  random 

number  tables). 

C821  Ability  to  interpret  and  integrate  statistical  data  into  a 

meaningful  presentation. 


C822 


Knowledge  of  alternate  methods  of  presenting  statistical 
data  (e.g..  charts,  graphs,  or  tables). 


C823  Ability  to  organize  and  classify  information  into  meaningful 

categories. 

C824  Knowledge  of  how  computers  might  be  used  to  analyze  data. 

C825  Ability  to  allocate  time  and  money  wisely  in  arranging 

computer  work. 

C826  Ability  to  design  card  layouts  to  allow  data  analysis  within 

computer  constraints  and  ability  to  use  standardized  computer 
programs  (e.g..  BMP  series). 

C827  Ability  to  write  computer  programs. 

C828  Knowledge  of  capabilities  of  local  computer  systems. 

C829  Ability  to  use  computer  tapes  or  discs. 

C830  Ability  to  use  computer  coding. 

C831  Ability  to  keypunch. 
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Competency 

Number  Competency  Categories 

C832  Ability  to  read  and  interpret  computer  output. 

C833  Ability  to  use  or  design  computer  simulation,  modeling,  or 

graphics. 

C834  Knowledge  of  philosophy  of  science. 

C835  Ability  to  use  computer-related  equipment  such  as  sorters. 

reproducers,  or  automatic  test  scoring  machines. 

C836  Knowledge  of  econometrics. 

C901  Knowledge  of  various  stages  in  the  research  process.  This 

competency  refers  to  knowing  what  tasks  must  be  engaged  in 
to  carry  out  a  research  project  (e.g.,  problem  delineation, 
collecting  data)  as  opposed  to  actually  knowing  how  to 
conduct  each  stage.  ^ 

C902  Knowledge  of  group  dynamics  or  skills  of  verbal  interaction. 

This  competency  r;efers  to  the  ability  to  lead  group  discussi ons , 
moderate  meetings  of  any  work  team,  and  facilitate  constructive 
interactions  among  personnel. 

C903  Knowledge  of  connotations  of  words.    This  competency  refers 

to  the  ability  to  avoid  attaching  an  unintended  connotation 
or  any  imprecise  meaning  to  words  in  either  oral  or  written 
communication. 

C904  Ability  to  identify  or  obtain  cooperation  from  needed 

resource  persons  or  consultants. 

C905  Ability  to  use  consultants  effectively.   This  includes  the 

ability  to  obtain  needed  Information  or  suggestions  fromf-^ 
consultants. 

C906  Ability  to  establish  workincj-SaTionships  with  other  groups 

or  agencies. 

C907  Ability  to  keep  up-to-date  with  new  developments  in  one's 

field  through  reading. 

C908  Ability  to  keep  up-to-date  with  new  developments  in  one's 

field  through  communication  with  other  persons  in  the  field. 


C909 


Knowledge  of  security  problems  (e.g. .  confidentiality) 
Involved  in  using  and  storing  Instruments  and  data. 
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Competency 
Number 


Competency  Categories 


C910  Knowledge  of  school  finance. 

C911  Ability  to  use  library  research  technigues.    This  competency 

refers  to  the  ability  to  locate  information  in  a  library 
and  would  Include  acquaintance  with  relevant  periodical 
indices. 

C912  Ability  to  use  ERIC  or  other  information  retrieval  systems. 

C913  Ability  to  develop  and  apply  an  effective  system  for 

summarizing  and  recording  information  obtained  through  a 
manual  search  of  library  sources  and  references!   This  skill 
relates  to  the  use  of  systematic  ways  to  record  notes ,  such 
as  index  cards  and/or  filing  systems. 

C914  Ability  to  operate  or  use  audio-visual  aids. 

C915  Knowledge  of  policies  and  procedures  of  local  ^  state,  or 

Federal  agencies  which  facilitate  or  support  inguiry  and 
inguiry-r elated  activities. 

C916  Knowledge  of  how  to  advertise  recently-developed  educational 

materials!   This  competency  includes  Knowledge  of  market 
research  techniques  and  the  ability  to  utilize  such  methods. 

C917  General,  speaking  skills.   This  competency  relates  to  the 

general  ability  to  communicate  orally  in  a  clear  and  effective 
fashion . 

C918  Ability  to  respond  extemporaneously  to  oral  questions. 

Included  would  be  the  general  ability  to  respond  to  questions 
in  any  situation  (e.g.,  a  TV  interview). 

C919  Ability  to  maintain  audience  interest  in  an  oral  presentation. 

C920  Ability  to  Vead  the  audience"  and  modify  an  oral  presenta- 

tion accordingly. 


cool  Ability  to  write.   This  is  the  general  ability  to  communicate 

written  Information  coherently.  - 

C002  Ability  to  revise  and  rewrite.    This  writing  skill  includes 

the  abilUy  to  refine  previous  writing,  deleting,  modifying, 
or  adding  written  material  where  necessary. 

C003  Ability  to  edit  one's  own  writing  or  that  of  others.  This 

competency  requires  the  ability  to  proofread,  a  knowledge  of 
gramnar  and  spelling,  and  the  ability  to  detect  errors  in 
gramnar  and  spelling. 

^  C004  Ability  to  synthesize  oV' summarize. 
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Number  Competency  Categories 


COOS  Ability  to  maintain  a  consistent  style  in  a  written  document. 

Consistency  of  style  would  include  maintaining  the  same 
level  of  language  difficulty,  the  same  degree  of  specificity, 
or  the  same  general  attitude  throughout  any  one  written 
document. 


C006 


C014 


Ability  to  determine  an  appropriate  scope  in  any  written  work. 
This  competency  includes  the  ability  to  determine  how  broad 
a  focus  to  take  in  any  writing  task. 


C007  Ability  to  write  macro-scenarios.   This  skill  might  be 

described  as  a  "story-writing  ability"  and  would  be  utilized 
by  someone  writing  curriculum  materials  or  certain  kinds  of 
historical  or  philosophical  analyses. 

C008  Knowledge  of  which  professional  journals  might  publish  a 

given  article. 


C009  Ability  to  put  quantitative  or  numerical  Information  into 

verbal  or  narrative  form.   This  skill  Involves  translating 
data  Into  meaningful  narrative,  as  in  reporting  a  summary 
of  findings  from  a  research  or  evaluation  study. 

COlO  Ability  to  write  in  a  style  and  at  a  level  appropriate  to 

a  specified  audience"    Tnis  competency  requires  (a)  a 
knowledge  of  the  audience's  point  of  view,  preferences, 
level  of  understanding,  etc.,  and  (b)  the  ability  to  use 
such  knowledge  in  choosing  an  appropriate  style  or  level 
of  coinnunicatlon  for  transmitting  information  to  that  audience. 

con  Ability  to  write  in  an  objective,  unbiased  manner.  This 

competency  involves  an  awareness  of  of  "sci en t i f i c" 

coinnunicatlon,  where  emphasis  is  placed  on  objectivity  and 
evidential  test. 

C012  Ability  to  write  for  practitioners.    This  skill  refers  to 

the  ability  to  write  in  a  non-technical  fashion,  perhaps 
purposely  avoiding  technical  terms,  in  order  to  communicate 
to  persons  such  as  public  school  teachers  or  administrators. 

C013  Ability  to  write  in  an  interesting  or  appealing  style.  This 

might  require  the  use  of  humor,  conflict,  or  any  techn 1 que 
which  increases  interest  without  sacrificing  communication. 


Ability  to  organize  ideas.   This  competency  refers  to  such 
aspects  of  writing  as  the  ability  to  outline,  the  ability 
to  focus  on  one  point  at  a  time,  or  the  ability  to  logically 
organize  elements  of  writing. 
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C015  Ability  to  write  clearly  and/or  concisely.    Included  in  this 

skill  would  be  a  knowledge  of  good  sentence  construction  and 
the  ability  to  use  words  and  phrases  which  coninunicate  clearly. 

C016  Ability  to  write  as  a  member  of  a  team. 

C017  Ability  to  describe,  explain,  or  elaborate  in  writing. 

C018  Knowledge  of  alternate  sources  of  funding  for  a  proposal  and 

reguirements  for  writing  proposals  appropriate  for  each 
source.*  This  competency  comes  into  play  when  one  is  wri ting 
a  proposal  to  obtain  funding  and  would  include  a  knowledge 
of  the  preferences  of  the  funding  agency  as  they  relate  to 
the  proposed  study  and  the  ability  to  use  written  communica- 
tion to  influence  the  funding  agency  to  approve  the  proposal  - 
i.e.,  "grantsmanship." 

C019  Ability  to  document  or  support  a  written  statement.  This 

competency  would  Include  not  only  the  ability  to  use  direct 
quotes  In  writing,  but  also  the  ability  to  use  bona  fide 
examples  In  backing  up  an  argunent. 


*Competency  018  may  also  be  categorized  under  several  other  general 
coin)etency  categories  since  it  Includes  much  more  than  simply  "writing  skills.' 
It  is  Included  here,  however,  in  the  first  relevant  category  which  is  dis- 
cussed.   Similarly,  other  competencies  which  mijght  fit  in  several  categories 
win  be  listed  in  the  first  relevant  category  which  appears  in  this  list. 
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TABLE  El 
Correlations  Among  Tasks 


TASK 


Tl        T2       T3  T4 


(630)     497      143  099 
(738)     026  277 
(689)  107 
(667) 


T5 

T6 

T7 

T8 

T9 

TIO 

Til 

T12 

121 

056 

045 

043 

139 

-087 

004 

-042 

086 

018 

-072 

012 

063 

035 

003 

-071 

091 

380 

061 

054 

033 

-079 

017 

054 

078 

-032 

-097 

-065 

-222 

013 

116 

-179 

(727) 

073 

-186 

569 

024 

-112 

186 

158 

(703) 

055 

032 

098 

-014 

004 

118 

(768) 

-142 

527 

392 

077 

-101 

(738) 

-023 

-034 

064 

150 

(780) 

316 

045 

-023 

(677) 

-071 

-089 

(699) 

107 

(693) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El 
Correlations  Among  Tasks 


TASK 


713 

T14 

T15 

T16 

T17 

T18 

T19 

T20 

T21 

T22 

T23 

T24 

Tl 

-043 

056 

-022 

-040 

108 

-057 

1  1  il 

-114 

111 
-111 

1  1  it 

114 

-055 

nil  ^ 

043 

01 1 

T2 

127 

-063 

-037 

-198 

095 

-134 

-126 

062 

035 

-083 

n^i 

-071 

-004 

T3 

-073 

183 

068 

013 

023 

Oil 

043 

-184 

nm 

091 

Ol  c 

21 6 

n*5i 

-031 

1  cn 

159 

T4 

023 

236 

055 

137 

089 

031 

004 

-024 

-112 

-108 

1  ^n 

-179 

-018 

T5 

205 

269 

281 

060 

-033 

002 

-072 

130 

353 

094 

203 

065 

To 

-010 

068 

179 

-101 

213 

-103 

-102 

-123 

120 

-026 

075 

099 

T7 
1  / 

-UO/ 

-079 

uio 

-uou 

U 1  u 

-919 
-c  1  C 

UO  1 

-in 

- 1  1  1 

-im 

—  1  U  1 

—  U  1  V 

T8 

249 

085 

227 

045 

-022 

-047 

025 

217 

327 

032 

103 

004 

T9 

067 

003 

048 

-211 

-048 

-022 

069 

-006 

009 

-086 

-023 

-032 

TIO 

-006 

-005 

137 

-007 

-095 

015 

-015 

216 

-094 

-017 

-034 

-030 

Til 

186 

200 

013 

290 

-063 

079 

-029 

-049 

220 

-080 

-150 

093 

T12 

295 

249 

390 

094 

-022 

160 

195 

078 

327 

226 

245 

323 

T13 

(623) 

150 

321 

060 

-033 

136 

091 

130 

109 

000 

021 

065 

T14 

(734) 

337 

228 

075 

258 

222 

089 

259 

165 

249 

179 

T15 

(629) 

086 

052 

103 

193 

264 

322 

190 

309 

217 

T16 

(616) 

094 

115 

042 

-025 

169 

018 

045 

022 

T17 

(569) 

076 

124 

-127 

104 

-013 

177 

-058 

T18 

(760) 

659 

009 

060 

-105 

-047 

027 

T19 

(791) 

097 

-042 

-051 

025 

073 

T20 

(672) 

207 

247 

171 

071 

T21 

(684) 

108 

159 

148 

T22 

(697) 

371 

261 

T23 

(712) 

243 

T24 

(670) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


TABLE  El 
Correlations  Among  Tasks 


TASK 


TASK 

1  CO 

1  ^0 

1  C/ 

1  CO 

TOO 
1  £7 

Ton 

T30 

TO! 

T31 

T32 

T33 

T34 

T35 

T36 

Tl 

070 

-021 

043 

-070 

007 

-054 

-061 

106 

-050 

-002 

079 

046 

T2 

-006 

-044 

0B2 

-096 

-019 

-126 

-075 

-017 

-054 

046 

006 

-036 

T3 

0B3 

-013 

-070 

-150 

-170 

-098 

-056 

110 

153 

050 

037 

024 

T4 

-077 

-112 

17B 

-093 

-203 

-104 

-114 

-068 

-119 

-143 

-045 

-147 

T5 

295 

27B 

049 

411 

153 

133 

065 

-099 

017 

-104 

-046 

013 

T6 

-049 

-04B 

002 

-133 

-107 

-084 

-192 

117 

259 

198 

206 

168 

T7 

147 

-066 

-110 

-166 

-045 

-197 

-019 

-136 

345 

252 

323 

424 

T8 

260 

313 

-061 

457 

291 

220 

084 

-093 

022 

021 

-021 

-059 

T9 

070 

-077 

-040 

-154 

-130 

-169 

042 

-108 

387 

152 

339 

488 

TIO 

-056 

-192 

-152 

-113 

-097 

-186 

-030 

-073 

189 

152 

259 

249 

Til 

175 

096 

-124 

220 

212 

167 

021 

-117 

092 

020 

045 

077 

T12 

043 

-077 

071 

058 

-070 

049 

004 

034 

106 

-071 

090 

-101 

T13 

130 

062 

-079 

170 

003 

078 

065 

024 

219 

073 

008 

013 

T14 

009 

-081 

166. 

-053 

-196 

-012 

110 

190 

025 

-167 

-032 

-251 

T15 

Oil 

-029 

-042 

022 

-067 

043 

-057 

171 

160 

-111 

067 

054 

T16 

-002 

005 

-042 

092 

-149 

018 

-144 

046 

-015 

-130 

-053 

-052 

T17 

036 

026 

-073 

-109 

005 

-089 

-058 

233 

-165 

063 

-092 

-004 

TIB 

029 

023 

-014 

-022 

-158 

-052 

-089 

132 

-068 

-131 

023 

-090 

T19 

05B 

091 

-087 

048 

-124 

-004 

-070 

187 

104 

-073 

091 

-138 

T20 

057 

126 

-203 

133 

-037 

032 

-007 

-097 

114 

-004 

017 

-091 

T21 

212 

170 

-082 

233 

136 

040 

077 

-Oil 

086 

-094 

-029 

-006 

T22 

02B 

053 

-184 

029 

001 

243 

342 

172 

141 

-047 

-061 

-111 

T23 

000 

090 

-061 

058 

085 

049 

243 

288 

-103 

-116 

-077 

023 

T24 

-042 

-024 

018 

-056 

-133 

079 

060 

170 

001 

-012 

258 

051 

T25 

(744) 

472 

-052 

471 

280 

185 

104 

-113 

122 

337 

216 

109 

T26 

(B52) 

-069 

662 

460 

293 

164 

-071 

013 

116 

-120 

-115 

T27 

(594) 

-105 

-094 

-062 

-093 

-056 

-146 

-143 

009 

-110 

T2B 

(851) 

566 

389 

280 

-084 

-045 

122 

-045 

-079 

T29 

(774) 

296 

388 

-080 

-056 

253 

-088 

-135 

T30 

(666) 

368 

239 

-069 

-032 

-045 

-097 

T31 

(768) 

170 

071 

065 

-118 

-158 

T32 

(572) 

-126 

-099 

-071 

-136 

T33 

(691) 

260 

261 

271 

T34 

(721) 

439 

173 

T35 

(768) 

421 

T36 

(776) 

NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estiamtes  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


368 


TABLE  El 
Correlations  Among  Tasks 


TASK 


T37  T38 


T39 


T40  T41 


TASK 
T42  T43 


T44  T45 


T46  T47 


T48 


Tl 
T2 
T3 
T4 
T5 
T6 
T7 
T8 
T9 
TIO 
Til 
T12 
T13 
T14 
T15 
T16 
T17 
T18 
T19 
T20 
T21 
T22 
T23 
T24 
T25 
T26 
T27 
T28 
T29 
T30 
T31 
T32 
T33 
T34 
T35 
T36 


005 
048 
■051 
-052 
-198 
023 
090 
-047 
042 
174 
079 
-116 
-064 
-041 
-015 
-100 
-069 
-005 
041 
076 
-124 
-034 
-047 
-089 
092 
104 
-014 
051 
218 
-052 
027 
-053 
176 
402 
104 
030 


-070 
168 
-061 
148 
-071 
-042 
095 
-142 
172 
175 
039 
-042 
074 
033 
065 
-Oil 
-109 
197 
048 
133 
010 
-124 
-042 
112 
059 
073 
-035 
052 
-032 
-093 
-056 
-084 
090 
074 
132 
269 


-204 
-140 
-061 
-093 
-023 
140 
182 
008 
358 
117 
-006 
008 
026 
-096 
-020 
-063 
-005 
051 
048 
036 
-034 
-073 
-142 
-056 
-033 
-103 
-035 
000 
-032 
-093 
028 
-084 
267 
267 
191 
357 


-008 
096 
-005 
-068 
-099 
233 
086 
034 
129 
-073 
-117 
-093 
024 
-141 
-157 
-086 
233 
-053 
-042 
-097 
-124 
-088 
-093 
-045 
004 
230 
-056 
050 
059 
086 
-045 
-027 
-014 
'  269 
-071 
086 


048 
025 
056 
-104 
-032 
124 
152 
-065 
150 
013 
-040 
-065 
-087 
-111 
-055 
-041 
-089 
-052 
-004 
-024 
-Oil 
068 
049 
079 
133 
158 
-142 
088 
047 
172 
079 
086 
184 
189 
158 
202 


-025 
080 
164 
148 
-119 
094 
052 
-192 
079 
060 
039 
-142 
-071 
-140 
-192 
-063 
-109 
-094 
-041 
-110 
-212 
-124 
-291 
-140 
-033 
-045 
-035 
-158 
-195 
-093 
-140 
-084 
178 
026 
073 
052 


-070 
-008 
-016 
208 
170 
003 
-035 
-042 
079 
-113 
-006 
-092 
074 
-010 
-020 
-063 
100 
124 
138 
-110 
-167 
-073 
008 
-056 
104 
250 
105 
105 
022 
027 
028 
-084 
-043 
-023 
014 
095 


-070 
-052 
-150 
-093 
170 
-133 
008 
207 
-061 
002 
129 
008 
026 
-096 
022 
041 
-109 
-167 
-131 
182 
099 
132 
058 
-056 
150 
309 
-035 
526 
512 
449 
364 
-084 
,134 
170 
-045 
-122 


153 
066 
016 
-018 
-166 
-047 
120 
004 
116 
-030 
021 
-075 
-089 
-028 
-193 
-061 
-058 
027 
073 
-085 
-136 
016 
-155 
060 
031 
-024 
018 
-056 
-046 
-018 
-075 
-045 
071 
219 
-024 
-019 


213 
020 
153 
-018 
-024 
374 
198 
064 
153 
092 
092 
106 
179 
025 
016 
-015 
100 
-068 
-046 
-050 
160 
055 
-062 
213 
160 
-036 
-087 
-043 
-010 
-018 
-141 
-126 
294 
300 
360 
235 


048 
-111 
146 
-101 
-178 
168 
162 
-103 
110 
106 
068 
-053 
-083 
-070 
-125 
-022 
-112 
-028 
-046 
-169 
038 
-033 
-053 
104 
-002 
005 
-042 
-Oil 
119 
077 
022 
046 
116 
155 
121 
119 


028 
-052 
099 
-042 
-187 
-020 
183 
-103 
064 
195 
-091 
041 
091 
-141 
-048 
137 
-066 
005 
-079 
-113 
-106 
055 
-031 
037 
052 
-133 
-004 
-083 
-150 
-042 
-084 
-051 
144 
091 
122 
057 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  ,used_as  estimates  of  coinmunality  in  the  factor  analysis. 
Leading  decimal  points, are  omitted. 


TABLE  El 
Correlations  Among  Tasks 


TASK 


TASK 

T49 

T50 

T51 

T52 

T53 

T54 

T55 

Tl 

161 

-122 

157 

-057 

Z04 

1  AA 

AOQ 

T2 

OBO 

-129 

073 

A^  A 

-013 

157 

131 

AOil 

T3 

252 

-120 

210 

-051 

193 

AAO 

AOO 

T4 

-076 

-068 

-047 

AP  A 

-052 

1  AA 

109 

1  1  A 

1  AC 

-100 

T5 

t196 

-099 

-106 

-131 

-079 

AOil 
VCH 

A01 

T6 

246 

-115 

093 

-103 

Zlc 

AAO 

T7 

255 

-136 

136 

-150 

-004 

OOil 

A1 A 

T8 

-155 

-093 

-142 

^  AP 

-185 

-254 

ACO 

T9 

140 

-108 

181 

-151 

AA^ 

097 

1  CO 

\)W 

TIO 

132 

-073 

106 

A^C 

-065 

1  A  A 

-IZZ 

007 

AOO 

Til 

-110 

-117 

039 

-171 

Ai  n 

-019 

ACA 

-lUO 

T12 

Oil 

161 

023 

AAl 

091 

A^A 

-032 

AOO 
VCC 

-uou 

T13 

046 

-099 

-027 

-064 

-036 

A70 

T14 

-172 

-141 

-072 

1  Al 

-101 

AO  A 

-020 

Ad 

AA1 

T15 

-073 

-047 

AAA 

-088 

-134 

-OZo 

1  DU 

T16 

-06B 

-086 

034 

AAA 

-028 

018 

AOQ 

10A 
-131 

T17 

OOB 

-035 

-177 

A^A 

-069 

138 

A77 

-0// 

-UD4 

TIB 

04B 

-053 

-150 

1  Af 

-106 

-048 

AA1 

MA 

"UOH 

— l/Ow 

-154 

-046 

087 

T20 

-019 

027 

070 

076 

-092 

073 

097 

T21 

-040 

-124 

-006 

-124 

135 

123 

034 

T22 

045 

172 

-069 

107 

-060 

-074 

036 

T23 

-071 

161 

-183 

022 

-076 

190 

110 

T24 

-004 

170 

-019 

027 

027 

071 

-070 

T25 

-006 

-113 

033 

-224 

-125 

006 

-097 

T26 

,006 

-071 

-017 

-140 

-140 

-135 

-010 

T27 

-093 

-056 

-052 

082 

096 

089 

032 

T2B 

-184 

-084 

008 

-094 

-230 

-043 

-041 

T29 

-110 

-080 

ood 

-158 

-218 

-010 

-031 

T30 

-126 

-068 

-097 

-052 

-160 

-018 

099 

^T31 

-145 

-045 

051 

-089 

-122 

-070 

073 

T32 

-017 

-027 

-136 

-053 

106 

-014 

-042 

T33 

242 

-126 

308 

-068 

067 

-003 

-122 

T34 

288 

-099 

173 

069 

-122 

-064 

-155 

T35 

203 

-071 

129 

023 

-140 

013 

-110 

T36 

219 

-136 

064 

-090 

151 

162 

-064 

T57      T58      T59  T60 


288  -040  -098  111  -014 

212      062  063  037  037 

299  -031  -126  038  059 

088  -101  -063  -073  -030 

-023  -035  024  189  -104 

140      033  -063  060  002 

095  -095  -050  087  -045 

-092  -053  021  090  -014 

079      110  096  051  -052 

060  -064  113  119  045 

-051      023  -035  -035  -004 

058  143  ,  065  184  136 
074  -035  -019  -  098  188 
163  -070  -035  148  076 
022      086  199  246  -020 

-063  -176  -120  -063  -075 

-005      094  230  -026  -062 

051  -100  -022  152  -095 

-041      042  148  103  -074 

-061      023  167  157  048 

099  -093  126  100  -019 

-022      118  049  -027  152 

108      045  1Q7  044  061 

028      022  116  -001  -080 

059  -137  -132  286  077 
-014  -111  -129  025  053 

035      027  -102  -002  -099 

-053  -Oil  -061  096  010 

-087      119  063  -028  188 

-154      137  -063  -073  061 

-056      104  -107  -001  302 

-084      046  010  -095  -048 

090     -059  -160  255  042 

-071      060  -061  144  042 

-045      005  '  -025  025  -126 

095      034  065  047  -111 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

,  correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading: decimal  points  are  omitted. 
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TABLE  El 
Correlations  Among  Tasks 


TASK 


TASK 

Tfil 

Tfi? 

Tfi4 

T65 

T66 

T67 

1  w  / 

T68 

T69 

Tl 

-054 

-157 

-163 

-099 

106 

-042 

-122 

-036 

-190 

T2 

098 

-177 

-121 

-105 

208 

144 

096 

-055 

-051 

T3 

046 

056 

-190 

043 

-005 

-039 

-120 

201 

033 

T4 

-025 

-035 

-077 

-055 

239 

188 

086 

006 

099 

T5 

088 

-087 

127 

-081 

-099 

-115 

-099 

-031 

009 

T6 

-120 

-084 

-207 

048 

-115 

-133 

-115 

216 

-102 

T7 

071 

252 

223 

-111 

-025 

-061 

-025 

-073 

-064 

T8 

071 

-122 

000 

080 

034 

003 

034 

053 

025 

T9 

047 

097 

192 

057 

-108 

-126 

-108 

062 

-089 

TIO 

187 

145 

195 

120 

074 

043 

074 

-085 

083 

Til 

009 

064 

214 

045 

-117 

-136 

-117 

034 

-029 

T12 

324 

-008 

-261 

-075 

034  . 

-108 

-093 

-Oil 

110 

T13 

186 

-032 

043 

-081 

024 

-008 

024 

-093 

091 

T14 

097 

087 

009 

-115 

-031 

-068 

-141 

-030 

075 

T15 

174 

-006 

Oil 

-127 

-047 

-087 

-047 

-012 

047 

T16 

-053 

077 

043 

-070 

046 

-100 

-086 

-122 

-046 

T17 

000 

-089 

-056 

-028 

-035 

-040 

-035 

-076 

-054 

T18 

133 

364 

029 

-043 

-053 

-062 

-053  • 

-023 

041 

T19 

236 

304 

136 

-034 

-042 

-048 

-042 

024 

239 

T20 

050 

-024 

100 

07Z 

not 

1  no 

1  Rl 

ini 

lUI 

T21 

027 

091 

-136 

-101 

-Oil 

-045 

-Oil 

071 

034 

T22 

197 

-108 

-061 

-072 

-088 

Oil 

042 

070 

036 

T23 

172 

-179 

000 

-075 

-093 

-108 

-093 

053. 

109 

T24 

034 

-018 

031 

-036 

-045 

-052 

-045 

226 

073 

T25 

028 

080 

123 

-092 

-113 

-131 

-113 

107 

-097 

T26 

-036 

023 

267 

126 

-071 

-082 

-071 

-078 

091 

T27 

-028 

-142 

-235 

-045 

123 

-065 

-056 

-122 

-087 

T28 

Oil 

--033 

150 

095 

-084 

-097 

-084 

-048 

048 

T29 

-077 

-078 

186 

274 

-080 

-092 

-080 

-034 

-124 

T30 

037 

-104 

080 

-055 

-068 

-079 

-068 

006 

099 

T31 

120 

-018 

178 

225 

-045 

-052 

-045 

010 

073 

T32 

-082 

-068 

-113 

-022 

-027 

-031 

-027 

114 

-042 

T33 

018 

184 

122 

-103 

-126 

-147 

-126 

065 

-046 

T34 

-108 

023 

169 

219 

024 

-115 

-099 

154 

-073 

T35 

024 

023 

164 

-057 

-071 

-082 

-071 

149 

-010 

T36 

-062 

-097 

109 

-111 

-136 

-158 

-136 

150 

-138 

NOTE: 


Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 
correlations  which  were  used  as  estimates  of  coiwnunality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


TABLE  El 
Correlations  Anong  Tasks 


TASK 

T37  T38 


(797)     197      197      132       114     -094     -094     -022      492      054      187  -100 


(733) 


T39 

T40 

T41 

T42 

T43 

T44 

T45 

T46 

T47 

197 

132 

114 

-094 

-094 

-022 

492 

054 

187 

210 

184 

208 

000 

158 

052 

-056 

-087 

041 

(684) 

184 

148 

000 

105 

-106 

028 

134 

092 

(665) 

239 

-084 

184 

050 

170 

099 

046 

(560) 

208 

148 

088 

079 

033 

256 

(645) 

000 

-106 

028 

090 

-063 

(612) 

-053 

-140 

-132 

-167 

(758) 

-056 

-087 

-115 

(729) 

142 

187 

(802) 

290 

(579) 

T48- 


-007 
-083 
-051 
045 
145 
-083 
-083 
-084 
208 
137 
(552) 


NOTE:  Correlations  are  phi  coefficients.  Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El 
Correlations  Among  Tasks 


TASK 


TASK 

T49 

T50 

T51 

T52 

T53 

T54 

TC  C 

T56 

TC7 
1  0/ 

1  Dy 

T37 

170 

-053 

090 

-005 

-113 

-068 

-083 

-022 

-100 

-087 

220 

T38 

-008 

-084 

008 

-167 

-089 

001 

-131 

158 

-oil 

-061 

195 

T39 

212 

-084 

-079 

051 

051 

GDI 

-131 

-211 

092 

-014 

047 

T40 

-017 

-027 

-136 

-053 

-133 

-126 

-042 

-084 

046 

-108 

-095 

T41 

025 

-068 

052 

-135 

-106 

-018 

099 

027 

-101 

-009 

153 

T42 

080 

-084 

182 

-094 

145 

-220 

-131 

000 

041 

-061 

-052 

T43 

-184 

-084 

-209 

-094 

-089 

-043 

138 

000 

-Oil 

-107 

145 

T44 

-184 

-084 

095 

-094 

-183 

134 

138 

000 

092 

-014 

-003 

T45 

136 

-045 

120 

027 

027 

-070 

-070 

-056 

022 

-181 

077 

T46 

538 

-126 

235 

115 

146 

146 

-197 

178 

-102 

-081 

172 

T47 

322 

-086 

291 

-028 

018 

072 

-046 

092 

-022 

-028 

-015 

T48 

329 

143 

120 

214 

065 

144 

-079 

069 

-163 

-137 

034 

T49 

(820) 

-017 

401 

291 

275 

168 

-126 

168 

-068 

-171 

078 

T5d 

(599) 

-136 

503 

-133 

-126 

-042 

-084 

-086 

010 

-095 

T51 

(709) 

151 

151 

089 

084 

182 

-009 

-127 

169 

T52 

(720) 

-048 

-068 

041 

-022 

-100 

-087 

015 

T53 

(682) 

186 

006 

098 

064 

-027 

069 

T54 

(754) 

254 

311 

072 

075 

048 

T55 

(583) 

-041 

042 

148 

103 

T56 

(608) 

-167 

-014 

145 

T57 

(655) 

384 

-063 

T58 

(729) 

-036 

T59 

(654) 

T60 

T61 

T62 

T63 

T64 

T65 

T66 

T67 

T68 

T69 

T60 

235 
010 
-070 
-048 
-030 
-070 
-070 
010 
-080 
042 
160 
140 
104 
-048 
285 
015 
-095 
109 
-074 
249 
003 
-052 
054 
(666) 


NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  conmiunality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


E.ll 

TABLE  El 
Correlations  Among 'Tasks 


TASK 


TASK 

T61 

T62 

T63 

T64 

T65 

166 

TCI 

To7 

TCD 

Too  ■ 

107 

T37 

-089 

-052 

029 

183 

132 

-062 

-053 

-023 

-083 

T38 

-096 

148 

059 

-068 

-084 

-097 

-084 

155 

048 

T39 

-203 

148 

059 

095 

050 

-097 

-084 

222 

048 

T40 

-082 

-068 

-113 

-022 

-027 

-031 

-027 

-058 

-042 

T41 

-025 

034 

185 

-055 

-068 

-079 

-068 

238 

-004 

T42 

Oil 

088 

150 

-068 

050 

019 

-084 

087 

-041 

T43 

064 

-033 

196 

-068 

-084 

-097 

-084 

-048 

048 

T44 

Oil 

-093 

196 

095 

-084 

-097 

-084 

-116 

048 

T45 

-051 

-114 

-042 

036 

170 

-052 

-045 

118 

-070 

T46 

-027 

-018 

-109 

-103 

-126 

-147 

-126 

178 

-046 

T47 

-053 

-041 

-002 

252 

-086 

-100 

-086 

211 

042 

T48 

'077 

045 

-014 

-041 

-051 

-059 

-051 

-013 

-079 

T49 

054 

-025 

-006 

032  ' 

-017 

-149 

-129 

171 

024 

T50 

055 

-068 

-113 

-022 

rD27 

-031 

-027 

-058 

-042 

TBI 

115 

052 

-005 

160 

-025 

-158 

-136 

038 

010 

TS2 

-051 

-052 

-161 

-043 

132 

-062 

-053 

-116 

041 

T53 

-057 

055 

-125 

086 

106 

019 

mil 
-014 

m 
1 1 1 

(VIA 

y  T54. 

108 

-018 

006 

035 

-126 

-147 

-126 

121 

-046 

T55 

236 

-004 

136 

-034 

-042 

-048 

-042 

-091 

239 

T56 

118 

027 

-079 

-06i3 

-084 

-097 

-084 

-048 

-131 

T57 

158 

-041 

-002 

091 

178 

130 

046 

078 

042 

TS8 

142 

-009 

192 

201 

129 

184 

129 

003 

148 

T59 

110 

153 

072 

-077 

031 

-110 

-095 

-018 

020 

T60 

178 

-121 

-062 

209 

-048 

-055 

-048 

-104 

061 

T61 

(556) 

-025 

121 

-067 

055 

024 

055 

-110 

236 

T62 

(654) 

080 

-055 

-068 

-079 

-068 

-149 

-004 

T63 

(744) 

193 

-113 

-130 

-113 

-Oil 

136 

T64 

(696) 

-022 

-025 

-022 

163 

-034 

T65 

(763) 

564 

315 

-058 

187 

T66 

(914) 

862 

-068 

150 

T67 

(881) 

-058 

187 

T68 

(616) 

024 

T69, 

(575) 

NOTE:  Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted.  / 
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TABLE  E2 
Competencies  in  Package  01 


COMPETENCY  NUMBER 


C201 

C223 

PR!  7 

C202 

C301 

CoUl 

C203 

C302 

C204 

C303 

C803 

C205 

C304 

C804 

C823 

C206 

C305 

C805 

C824 

C207 

C306 

C806 

C825 

C208 

C3G7 

C807 

C826 

C209 

C308 

C808 

C827 

C210 

C309 

C810 

C828 

C211 

C313 

C811 

C830 

C212 

C314 

C812 

C831 

C215 

C701 

C813 

C832 

C217 

C702 

C814 

C834 

C218 

C726 

C815 

C911 

C221 

C728 

C816 

C912 

E.13 


TABLE  E3 
Competencies  In  Package  02 


COMPETENCY  NUMBER 


C101 

C403 

Cd09 

C90o 

C102 

C405 

CoiO 

C910 

C103 

C40d 

Coll 

C911 

Ail 

CI  04 

C407 

Col  3 

C91Z 

CIOd 

C40o 

Co  14 

C9I0 

CI  06 

C409 

Colo 

C917 

C107 

C413 

PCI  c 

Colo 

COOl 

1  .  ■ 

CI  08 

C414 

0617 

0002 

CI  09 

C415 

0620 

0003 

f 

Clio 

C416 

0622 

0004 

cm 

C417 

0625 

0009 

C112 

C418 

0626 

0010 

cn3 

C420 

0627 

0011 

C1 14 

C421 

0628 

0012 

i 

C115 

C603 

0629 

0013 

C116 

C604 

0630 

0014 

C118 

C605 

0902 

0015 

C119 

C606 

0903 

0017 

C401 

C607 

0904 

0018 

C402 

C608 

0905 

0019 

E.14 


TABLE  E4 
Cotn;)etenc1es  in  Package  03 


COMPETENCY  NUMBER 


C310 

C405 

C541 

0620 

0917 

C312 

C406 

C542 

C622 

0001 

C361 

C407 

C543, 

C625 

0002 

C363 

C408 

C545  , 

C626 

0003 

C364 

C547 

C627 

0004 

C366 

C411 

C603 

C628 

0005 

C368 

C412 

C604 

C629 

0006 

C371 

C413 

C606 

C630 

C009 

C372 

C414 

C608 

C905 

0010 

C373 

C415 

C610 

C907 

coil 

C374 

C418 

C611 

C908 

0012 

C375 

C421 

C613 

C910 

0013 

C376 

'C501 

C614 

C9n 

0014 

C378 

C502 

C615 

C912 

0015 

C401 

C506 

C616 

C914 

0016 

C402 

C507 

C617 

C915 

0017 

C404 

C508 

C618 

C916 

0018 

E.15 
TABLE  E5 

Correlations  Among  Package  01  Competencies 


COMPE- 
TENCY 

C201 
C202 
C203 
C204 
C205 
C206 
C207 
C208 
C209 
C210 
C211 
C212 
C215 
C217 
C218 
C221 
C223 
C301 
C302 
C303 
•X304 
C305 
C306 
C307 
C308 
C309 
C313 
C314 
C701 
C702 
C726 
C728 
C729 
C801 
C802 
C803 


COMPETENCY 


(696)    ^041  055 
(686)  323 
(587) 


C204 

C205 

C206 

C207 

C208 

C209 

C210 

C211 

218 

202 

144 

166 

-034 

-148 

046 

317 

269 

177 

168 

336 

197 

031 

-030 

247 

060 

015 

-027 

146 

-094 

-020 

046 

086 

(703) 

217 

075 

301 

259 

095 

038 

341 

(669) 

219 

091 

015 

-105 

-068 

229 

(721) 

374 

431 

015 

-014 

076 

(658) 

374 

137 

089 

187 

(814) 

263 

046 

008 

(771) 

221 

-112 

(673) 

173 

(797) 

NOTE:   Cbrrelatlonis  are  phi  coefficients.    Diagonal  elements  are  squaried  multiple 

correlations  Which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis. 
Leading  dec  1  ma V  points  are  oral t^^^ 
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TABLE  E5 

Correlations  Among  Package  01  Competencies 


COMPETENCY 


COMPt- 
TENCY 

C212 

C215 

C217 

C218 

C221 

C223 

C301 

C302 

C303 

C304 

C305 

OCT 

Obi 

1 09 

ififi 

125 

-094 

145 

ICC 
IDD 

AOO 

-Ooo 

u/u 

991 

1 90 
1  CM 

9A7 

-030 

0^1 

236 

063 

CZ03 

-OZO 

n9c 

— U*rU 

1  U9 

-197 
-lb/ 

O^R 

W  1  o 

017 

VI/ 

CZ04 

9AA 

OKA 

9K7 

9AQ 

USD 

1 AA 

Od^ 

211 
fall 

238 

115 

134 

-005 

215 

185 

287 

375 

246 

149 

211 

C206 

330 

272 

038 

058 

246 

131 

141 

204 

074 

224 

165 

C207 

203 

250 

176 

029 

349 

149 

133 

029 

000 

369 

256 

C208 

169 

141 

198 

307 

196 

133 

086 

133 

-035 

244 

291 

C209 

106 

-009 

135 

295 

-021 

-004 

-112 

146 

-031 

080 

008 

C210  -  ■ 

-004 

062 

019 

104 

-118 

311 

-076 

-034 

042 

074 

181 

C211 

361 

353 

197 

048 

142 

297 

103 

-014 

-076 

178 

193 

C212 

(679) 

189 

309 

146 

194 

146 

023 

146 

092 

192 

186 

C215 

(795) 

431 

153 

307 

153 

188 

-120 

056 

210 

181 

C217 

(679) 

099 

232 

019 

052 

-141 

065 

306 

234 

C218 

(786) 

014 

104 

-076 

-034 

155: 

-030 

181 

C221 

(589) 

014 

202 

-052 

121 

336 

092 

C223 

(665) 

-076 

-034 

042 

126 

126 

C301 

(629) 

110 

127 

131 

095 

C302 

(683) 

099 

022 

126 

C303 

(600) 

064 

178 

C304 

(666) 

514 

C305 

(772) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communallty  In  the  factor  analysis 
Leading  decimal  points  are  omitted. 


TABLE  E5 

Correlations  Among  Package  01  Competencies 


COMPETENCY 


COMPE- 
TENCY 

C306 

C307 

C308 

C309 

C313 

C314 

C701 

0702 

C726 

C728 

C729 

C201 

033 

1  Ail 

104 

1  il  C 

145 

Uo2 

noi 
-Uol 

-Ol  K 
-Ul  D 

079 
U/ C 

-010 
-U  1  u 

C202 

223 

An 

-057 

1  il  o 

142 

u/u 

-UoO 

.ni  A 

009 

1  77 

UOO 

C203 

1  OA 

128 

A  AC 

-005 

1  Ol 

-121 

1  70 

1/2 

ncQ 
-UM 

1  C7 
- 1  D/ 

-UoH 

VIVO 

-UOH 

01  5 

/^O  Ail 

C204 

180 

11"? 
117 

Ol  1 

ino 

-U2I 

ORA 
U3D 

917 

—UOO 

126 

C205 

168 

Al  f% 

-019 

A£n 

060 

U22 

noi 

1  07 
1 0/ 

loD 

-OAO 

-UHU 

-144 

C206 

160 

1  Oil 

184 

Zee 

Ol  c 

1 

-u/u 

OAQ 
UD^ 

UOD 

^ni 

OU  1 

-118 

C207 

1  AA  ' 

108 

159 

1  Cl 

161 

lo2 

.ORl 
-UDI 

1 

1 DD 

1 A7 

1Q7 

222 

-081 

C208 

n  OA 

128 

1  ii  f 
146 

360 

302 

071 
U/  1 

1  AA 

01  R 
U 1  9 

9A0 

ni  ^ 

U  1  9 

01  R 

U  1  9 

C209 

A^  A 

078 

1 

137 

1  oc 
loD 

U/2 

n9Q 
-U2y 

007 
UU/ 

0fi9 
Uo£ 

1 AQ 

^17 
0  1  / 

1  07 

C210 

162 

AAA 

089 

AAO 

-092 

no  7 
-03/ 

004 

OOl 

1  fifi 
1  OO 

U30 

-017 

-U  1  / 

UHO 

133 

C211 

147 

1  A? 

187 

AAC 

095 

1  Ol 

121 

991 

-221 

024 

1  OK 
1  Cq 

iik; 

0A9 

UHb 

1  03 

OAS 

UOO 

C212 

078 

Of  A 

,  268 

1  Of 

127 

A"f  O 

072 

n9Q 

007 

UU/ 

005 

1  AO 
1 

919 

1 AQ 

1  OZI 

16Q 

1  07 

C215 

A  A^ 

226 

OA  A 

329 

A  Of 

326 

OAO 

203 

i  9ii 
124 

Ol  1 

Ul  1 

O90 

-U2o 

115 

^0^ 

ouo 

•UO«r 

C217 

^  AA 

188 

A^  ^ 

316 

OAf 

297 

1  to 

178 

-005 

1  7C 
1  /D 

OCA 

1  o*^ 

9A^ 
£f  0 

AOO 
HUU 

IQfi 

C218 

A  t  ^ 

-037 

AAA 

089 

236 

Of  o 

273 

-ObO 

1  ilO 

i^y 

09A 

-UUD 

OfiO 
UOU 

1  -J  u 

K^CC  1 

1 

997 

168 

-053 

www 

228 

094 

247 

279 

-053 

— uo/ 

Iftl 
1  O  1 

015 

-125 

-031 

WW  1 

188 

-005 

080 

046 

046 

Uou  1 

1  Q9 

— uuo 

168 

187 

024 

w*»  • 

-066 

-056 

-045 

008 

-149 

113 

-031 

181 

015 

198 

149 

046 

058 

-113 

-127 

-127 

C303 

102 

-049 

045 

263 

000 

049 

058 

-013 

079 

177 

106 

C304 

335 

369 

'350 

242 

-075 

-039 

094 

149 

172 

178 

-018 

C305 

249 

351 

266 

336 

-077 

161 

033 

010 

-016 

154 

017 

C306 

(691) 

282 

249 

206 

097 

152 

033 

168 

058 

316 

065 

C307 

(722) 

256 

357 

105 

054 

043 

202 

281 

146 

071 

C308 

(§79) 

173 

-026 

161 

-023 

-040 

213 

223 

017 

C309 

(715) 

-035 

042 

192 

139 

094 

172 

041 

C313 

(641) 

049 

016 

-096 

056 

-059 

-140 

C314 

(565) 

-019 

-019 

-139 

071 

071 

C701 

(756) 

331 

187 

055 

-123 

C702 

(709) 

137 

414 

175 

e726 

(756) 

159 

159 

C728 

(799) 

343 

C729 

<  ■ 

(792) 

C801 

C802 

Ci303 

NOTE:  Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
.      Leading  decimal  ppints  are  omitted. 
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TABLE  E5 

Correlations  Among  Package  01  Competencies 


COMPE- 
TENCY 

C201 

C202 

C203 

C204 

C205 

C206 

C207 

C208 

C209 

C210 

C211 

C212 

C215. 

C217 

C218 

C221 

C223 

C301 

C302 

C303 

C304 

C305 

C306 

C307 

C308 

C309 

C313 

C314 

C701 

C702 

C726„ 

C728 

C729 

C801 

C802 

C803 


COMPETENCY 


C801 

C802 

C803 

C804 

CoOS 

rone 

l/oU/ 

316 

005 

041 

263 

154 

016 

462 

252 

-078 

201 

176 

159 

334 

072 

119 

-114 

197 

185 

092 

086 

-035 

221 

-020 

231 

141 

311 

293 

301 

447 

-037 

085 

207 

020 

046 

349 

232 

173 

115 

012 

-001 

270 

252 

244 

072 

074 

107 

269 

291 

295 

119 

263 

071 

-013 

162 

378 

248 

-083 

-056 

-023 

-083 

075 

036 

031 

-022 

071 

070 

-022 

025 

-070 

211 

282 

091 

111 

141 

404 

246 

279 

201 

269 

-023 

087 

075 

288 

153 

286 

090 

100 

216 

335 

203 

205 

193 

048 

-038 

132 

179 

059 

196 

083 

071 

120 

-022 

080 

239 

042 

297 

051 

078 

147 

312 

212 

282 

030 

-004 

170 

187 

190 

162 

099 

377 

-044 

066 

188 

006 

037 

229 

187 

071 

070 

030 

-030 

007 

155 

288 

-092 

-051 

AAA 

288 

A  Ail 

-034 

-016 

I  CO 

389 

192 

047 

350 

201 

218 

445 

488 

245 

063 

157 

194 

270 

445 

353 

187 

186 

239 

148 

103 

305 

198 

333 

-057 

-030 

124 

156 

197 

157 

127 

063 

157 

150 

148 

178 

374 

184 

220 

139 

230 

152 

306 

079 

111 

Oil 

030 

034 

-017 

000 

-030 

072 

-188 

-030 

-116 

-047 

000 

-006 

005 

-Oil 

-006 

154 

175 

173 

-082 

-174 

085 

-034 

070 

117 

090 

033 

108 

-077 

033 

220 

227 

000 

119 

-114 

071 

053 

231 

086 

248 

-078 

-114 

-055 

-013 

092 

-012 

-035 

(785) 

257 

063 

325 

172 

226 

459 

(737) 

-023 

030 

075 

204 

153 

(507) 

025 

159 

166 

-010. 

081 1     081 2 


221 
095 
-041 
327 
182 
007 
136 
310 
023 
034 
166 
023 
344 
187 
312 
271 
145 
066 
-022 
159 
175 
296 
044 
184 
208 
288 
043 
039 
106 
-073 
"150 
169 
-041 
229 
204 
-026 


281 
Oil 

-059 
286 
082 
028 
105 
104 
041 
004 
128 

-099 
294 
070 
198 
168 
069 
070 
004 
053 
070 
127 
143 
218 
178 
158 
154 
105 
016 

-155 
056 
104 
022 
275 
251 
-129 


466 
-023 
-114 
152 
238 
173 
072 
075 
-137 
146 
158 
188 
189 
135 
071 
265 
295 
091 
-004 
092 
249 
186 
187 
203 
127 
184 
-029 
-124 
236 
-037 
212 
169 
-114 
257 
188 
031 


164 
100 
141 
223 
031 
-016 
170 
026 
090 
-029 
271 
-009 
277 
007 
062 
133 
244 
106 
062 
-019 
210 
326 
094 
250 
181 
203 
-047 
-069 
071 
-053 
-079 
141 
026 
286 
189 
r033 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  cornnunallty.ln  the  factor  analysis. 
Leadlng-declmail  points  are  omitted. 
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TABLE  E5 

Correlations  Among  Package  01  Competencies 


1 


COMPE- 
TENCY 

C201 

C202 

C203 

C204 

C205 

C206 

C207 

C208 

C209 

C210 

C211 

C212 

C215 

C217 

C218 

C221 

C223 

C301 

C302 

C303 

C304 

C305 

C306 

C307 

C308 

C309 

C313 

C314 

C701 

C702 

C726 

C728 

C729 

C801 

C802 

C803 


C813    C814    C815  C816 


COMPETENCY 
081 7     081 9  C821 


C822     C823     C824  C825 


075 

273 

194 

424 

165 

153 

373 

340 

109 

118 

202 

109 

374 

282 

291 

197 

233 

150 

060 

-035 
224 
208 
054 
122 
253 
213 
019 

-030 
253 
059 
172 
194 
049 
367 
109 
064 


043 
205 
146 
347 
091 
247 
264 
374, 
072 
029 
187 
072 
250 
246 
210 
176 
089 
079 
029 
098 
278 
256 
065 
212 
256 
357 
-064 
-103 
166 
091 
113 
146 
071 
427 
137 
-014 


236 
-023 
-020 
381 
376 
015 
137 
169 
025 
-004 
158 
188 
288 
309 
445 
194 
071 
091 
146 
153 
136 
304 
241 
137 
363 
240 
-029 
137 
159 
101 
004 
358 
-020 
314 
106 
031 


275 
143 

-123 
164 
072 
218 
166 
412 

-071 

-024 
125 
159- 
164 
275 
046 
296 
188 
125 

-024 
058 
253 
257 
085 
166 
201 
192 
016 

-019 
130 

-123 
088 
055 

-123 
316 
312 

■Oil 


312 
099 
038 
227 
170 
119 
075 
038 
-086 
024 
279 
060 
410 
245 
092 
389 
092 
157 
024 
247 
157 
215 
188 
193 
215 
401 
057 
075 
-033 
046 
071 
293 
038 
269 
060 
099 


240 
137 
232 
227 
047 
002 
226 
001 
-052 
038 
145 
148 
341 
115 
161 
189 
038 
090 
-023 
-037 
206 
274 
260 
333 
128 
195 
174 
066 
052 
-009 
-050 
155 
-076 
402 
479 
092 


289 
114 
128 
218 
168 
265 
282 
191 
-084 
-087 
282 
187 
292 
072 
162 
213 
063 
102 
063 
020 
260 
288 
171 
282 
209 
244 
143 
-110 
-069 
-061 
197 
191 
-061 
315 
349 
042 


227 
055 

-068 
168 
179 
250 
035 
200 

-128 
046 
305 
160 
371 
085 
152 
115 
312 
065 

-060 
054 
221 
183 
120 
221 
183 
208 

-201 
035 
064 

-016 
192 
133 

-135 
235 
333 
055 


-Oil 
048 

-153 

-006 

-009 
002 

-147 
155 

-052 
161 

-020 

-052 
179 
044 
038 
072 
161 
090 

-084 
012 
068 

-113 
127 
013 
128 

-034 
059 

-040 

-Oil 
103 
206 
155 
001 

-108 
015 
048 


043 
161 

-081 
163 
368 
247 
054 
222 

-059 
029 
133 
072 

-069 

-034 
270 

-054 
270 
024 
149 
049 
097 
066 
021 

-156 
209 
042 

-120 
054 
104 

-019 
029 

-005 

-157 
198 
072 
117 


-101 
244 
051 
-013 
069 
118 
139 
180 
-094 
-003 
154 
240 
-074 
-086 
100 
-014 
100 
-030 
-003 
021 
-023 
006 
-057 
-040 
-075 
050 
-116 
-129 
109 
-025 
074 
051 
-078 
137 
351 
021 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared'multiple 

correlations  whlth  were  used  as  estimates  of  communal ity  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 


E.20 


TABLE  E5  - 
Correlations  Among  Package  01  Competencies 


COMPETENCY 


C826 

C827 

C828 

C830 

C831 

C832 

C834 

C911 

C912 

UoD 

1  D  1 

lit 

069 

164 

-113 

-050 

1  DD 

U/  0 

245 

115 

166 

-162 

-118 

m  0 

Uoo 

065 

026 

-181 

-030 

000 

1  nc 

1 UD 

\  OH 

oou 

075 

084 

-075 

-248 

000 

uoy 

n^A 

I  Of 

-053 

085 

049 

1  ii 

Uoy 

flQQ 

IQO 

158 

-112 
lit- 

010 

-097 

1  HO 

1  of 

-flRA 

134 

247 

oil 

-057 

-039 

^  1  J 

UOD 

Al  ? 

277 

339 

-089 

-055 

-101 

noo 

.no? 

-086 

173 

-009 

031 

-085 

1  1Q 
-  1  10 

OAR 

-043 

-014 

062 

020 

-006 

1  Oc, 

9C\9 
CKIC 

1 07 

400 

141 

271 

-250 

-222 

1  90 

mo 
-U  /c 

-Ut  1 

907 
^\jf 

173 

-009 

-078 

-085 

1  CI 
1  Dl 

I  Oo 

U/  1 

932 

-016 

036 

034 

062 

uy/ 

n9i 

n7Q 

97*5 

167 

038 

113 

-038 

007 

HA  7 

uou 

117 

996 

276 

-029 

-030 

-120 

1  HQ 

nQ7 
uy/ 

1 1Q 

1  Rl 

196 

177 
tit 

046 

078 

-032 

u/y 

nA7 

uou 

1  00 

996 

058 

-029 

020 

-063 

UOD 

-00? 

036 

010 

-058 

-024 

087 

139 

-047 

-052 

046 

092 

058 

-120 

220 

051 

085 

-024 

-148 

058 

-082 

015 

-093 

153 

047 

21  f 

197 

-Oil 

147 

157 

224 

004 

010 

-136 

333 

223 

040 

312 

162 

107 

-035 

-016 

-123 

132 

198 

165 

085 

091 

-050 

028 

114 

037 

198 

185 

119 

228 

193 

184 

-069 

-057 

-139 

286 

038 

092 

145 

268 

165 

-035 

023 

-033 

293 

186 

030 

245 

300 

270 

-070 

-080 

-359 

-184 

-136 

-079 

016 

-070 

-108 

-047 

245 

276 

095 

-070 

004 

-019 

-043 

-133 

091 

205 

010 

034 

205 

-109 

130 

174 

144 

071 

•062 

-167 

-083 

-038 

-148 

-058 

-078 

-048 

031 

-099 

-161 

023 

017 

-029 

088 

165 

098 

-079 

-007 

085 

138 

130 

-053 

144 

038 

065 

026 

071 

-030 

-086 

-165 

-136 

-034 

-132 

065 

026 

-055 

-101 

451 

253 

147 

263 

320 

177 

009 

-051 

-127 

321 

118 

122 

236 

280 

252 

-107 

085 

-085 

-030 

-046 

-018 

041 

099 

010 

-033 

020 

-118 

NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 
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TABLE  E5 

Correlations  Anong  Package  01  Competencies 


COMPE- 
TENCY 

C804 

C805 

C806 

C807 

C808 

C810 

C811 

C812 

C813 

C814 

C815 

C816 

C817 

C819 

C821 

C822 

C823 

C824 

C825 

C826 

C827 

C828 

C830 

C831 

C832 

C834 

C911 

C912 


C801     CjO'"'  C803 


COMPETENCY 


(638)     214  050 
(766)  341 
(655) 


246 
146 
110 
(801) 


137 
489 
165 
341 
(765) 


CBIO 

C811 

C812 

079 

201 

147 

240 

135 

480 

-017 

036 

203 

264 

397 

056 

356 

265 

270 

(668) 

181 

465 

(822) 

090 

(837) 

S^I^mUJI™!  'I-  E"^  "efficients.   Diagonal  elements  are  squared  multlDle 
S?;rSr?5i:;?rarj|tl5!^""*"  of  co™„nal1ty  1„\he'  'fartiJ'atlysis. 


884 


E.22 


TABLE  E5 

Correlations  Pmong  Package  01  Competencies 


C826 
C827 
C828 
C830 
C831 
C832 
C834 
C911 
C912 


COMPETENCY 


COMPE- 
TENCY 

C8I3 

C814 

C815 

C816 

C817 

C819 

C821 

C822 

C823 

C824 

C825 

C804 

148 

061 

144 

102 

064 

031 

277 

235 

124 

107 

-096 

C805 

26/ 

221 

075 

154 

387 

139 

387 

282 

139 

-020 

Oil 

C806 

257 

291 

036 

175 

074 

235 

215 

243 

-040 

156 

140 

C807 

247 

197 

336 

231 

302 

212 

265 

271 

-037 

197 

021 

C808 

327 

233 

386 

278 

345 

151 

286 

341 

200 

136 

181 

C810 

235 

161 

251 

148 

246 

345 

330 

247 

002 

049 

-020 

C811 

109 

007 

431 

466 

207 

214 

295 

391 

148 

203 

128 

C812 

222 

250 

090 

164 

232 

422 

358 

301 

01 7 

-069 

062 

C813 

(813) 

726 

297 

253 

102 

289 

137 

171 

033 

222 

203 

C814 

(806) 

137 

289 

134 

173 

108 

174 

066 

159 

228 

C815 

(882) 

082 

280 

280 

295 

218 

-052 

333 

128 

C816 

(813) 

243 

240 

136 

227 

052 

043 

214 

C8i7 

(760) 

205 

188 

081 

-034 

-160 

-009 

C819 

(782) 

304 

191 

-139 

-040 

146 

C821 

(696) 

465 

039 

195 

165 

C822 

(725) 

238 

360 

228 

C823 

(626) 

066 

-035 

C824 

(686) 

317 

C825 

(744) 

NOTi::    Correlations  are  ph1  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communallty  In  the  factor  analysis 
Leading  decimal  points  are  omitted. 
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TABLE  E!3 

Correlations  Among  Package  01  Competencies 


COMPE- 
TENCY 

rood 
Loco 

POOO 

Lo^o 

no  on 

toJU 

COMPETENCY 
C831  C832 

to  .5  4 

\y3  1  1 

C804 

136 

-013 

-103 

048 

013 

17/ 

-060 

025 

003 

C805 

062 

189 

?06 

154 

cZo 

-030 

039 

024 

C806 

080 

137 

212 

254 

0/0 

o  *7  n 
ci\J 

-103 

-002 

-137 

C007 

472 

215 

013 

231 

O  C"  o 

252 

056 

-010 

-140 

C808 

409 

107 

111 

1C6 

O  /I  IT 

345 

242 

-098 

095 

081 

C810 

371 

247 

291 

016 

183 

o  o  o 

0Z8 

038 

105 

-044 

C811 

256 

245 

194 

236 

280 

173 

090 

-023 

038 

C812 

240 

462 

394 

164 

232 

O  *7  O 

036 

-033 

-088 

C813 

356 

095 

032 

312 

440 

21  3 

-083 

-062 

-120 

C814 

353 

083 

004 

351 

486 

311 

-148 

-101 

-189 

C815 

514 

118 

122 

236 

907 

1  /  J 

-009 

-023 

-023 

C316 

216 

145 

228 

493 

519 

144 

164 

-062 

-050 

C817 

123 

112 

261 

105 

211 

-023 

054 

147 

-075 

C819 

265 

407 

364 

177 

265 

195 

179 

003 

-078 

C821 

304 

198 

117 

08b 

237 

213 

-103 

-031 

-004 

C822 

478 

187 

217 

227 

289 

306 

-120 

-061 

-107 

C823 

-050 

-112 

-103 

-Oil 

085 

-063 

-144 

225 

225 

C824 

405 

032 

119 

228 

251 

247 

-148 

-057 

-039 

C825 

395 

211 

083 

319 

491 

442 

-074 

-202 

-058 

C826 

(850) 

271 

221 

338 

470 

423 

-074 

-202 

-198 

C827 

(704) 

488 

145 

226 

224 

230 

-216 

-158 

C828 

(713) 

228 

325 

177 

133 

-066 

-032 

C83G 

(779) 

519 

442 

-023 

-165 

-226 

C831 

(802) 

403 

054 

-146 

-186 

C832 

(778) 

-112 

-201 

-217 

C834 

(538) 

-099 

-013 

C911 

(676) 

502 

C912 

(659) 

NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.24 
TABLE  E6 

Correlations  Among  Package  02  Competencies 


CI  08 

- 

CI  09 

Clio 

cm 

CI  12 

-020 

024 

-111 

001 

017 

168 

269 

306 

051 

165 

150 

188 

094 

240 

116 

082 

159 

122 

091 

098 

197 

119 

265 

214 

-020 

363 

130 

390 

131 

-010 

236 

230 

322 

306 

085 

(760) 

156 

274 

130 

098 

(761) 

184 

258 

000 

(820) 

177 
(738) 

209 
039 
(774) 

COMPE 
TENCY 


COMPETENCY 


CI  01      CI  02 


ClOl 
CI  02 
CI  03 
C104 
CI  05 
CI  06 
C107 
CI  08 
CI  09 
Clio 

cm 

C112 

C113 

C114 

C115 

CI  16 

C118 

C119 

C401 

.C402 

C403 

C405 

C406 

C407 

C408 

C409 

C413 

C414 

C415 

C416 

C417 

C418 

C420 

C421 

C603 

C604 


(631)     057     -019  -020 
(778)     125  ^086 
(730)  228 
(722) 


-022 
231 
167 
065 

(773) 


-137 
243 
155 

-057 
367 

(768) 


096 
057 
168 
125 
099 
165 
(727) 


E^^  coefficients.   Diagonal  elements  are  squared  multiple 
SirL°"cL^rp^oi^t\^^^^^^^  Of  co^unallty  inV  factor  Snalysis. 
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COMPE- 
TENCY 


TABLE  E6 

Correlations  Among  Package  02  Competencies 


COMPETENCY 


C113 

C114 

C115 

C116 

C118 

C119 

C401 

C402 

C403 

C405 

C406 

C407 

061 

051 

-lOb 

-0/0 

-004 

-UOt 

-076 
— u  /  u 

168 

-006 

185 

294 

213 

318 

198 

1  07 

lo/ 

322 

ni  ft 

U  1  o 

058 

-061 

VW  1 

161 

096 

/X  p  O 

-052 

Ail  ^ 

043 

iDD 

-081 

mil 

-076 

-UDO 

105 

065 

156 

268 

-007 

-165 

-023 

-0^0 

-U4U 

n09 

-020 

u  /  o 

-030 

014 

241 

055 

219 

209 

Ol  A 

219 

OOA 

Zo9 

-U/ 1 

inft 
lUo 

049 

-122 

013 

w  1  W 

194 

115 

286 

293 

165 

170 

1  OVI 

- 1^4 

-u^o 

-115 

-160 

026 

w  ^  w 

151 

226 

096 

190 

AAil 

-004 

-08^ 

inQ 

UU4 

ion 

-082 

062 

IbU 

one 

mo 

lUO 

97n 
^/u 

1  Hi? 

-187 

070 

101 

-137 

017 

-144 

162 

263 

299 

050 

161 

195 

-132 

-022 

141 

013 

-016 

-132 

121 

207 

322 

322 

013 

273 

-088 

177 

240 

-088 

-120 

020 

101 

152 

230 

295 

077 

163 

-123 

-132 

-009 

101 

058 

298 

-031 

-045 

-119 

-135 

-051 

105 

119 

146 

225 

070 

025 

186 

(773) 

330 

-030 

272 

151 

240 

009 

-044 

033 

062 

-112 

-004 

(756) 

138 

258 

138 

334 

-163 

-117 

026 

-125 

-055 

030 

(800) 

323 

297 

289 

-108 

071 

057 

'069 

-082 

-121 

(851) 

057 

268 

-155 

-053 

-055 

-107 

-113 

-014 

(665) 

042 

033 

137 

-076 

-069 

069 

-060 

(715) 

-037 

-153 

-059 

-067 

-030 

-040 

(699) 

138 

126 

190 

207 

147 

(732) 

211 

142 

-044 

031 

(780) 

050 

-014 

-014 

(769) 

207 

342 

(752) 

185 

(735) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  coninunality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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r 

TABLE  E6 

Correlations  Among  Package  02  Competencies 


COMPETENCY 


utuo 

Pdl  4 

UH  1  ^ 

Pdl  7 

Pdl  ft 

Pd9n 

Pd91 

ClOl 

161 

101 

309 

083 

215 

-030 

110 

-030 

083 

018 

-013 

051 

CI  02 

046 

-089 

-121 

100 

-126 

001 

-058 

001 

-026 

265 

037 

270 

C103 

-025 

048 

-040 

-058 

024 

-016 

-302 

040 

182 

298 

002 

166 

C104 

-039 

175 

122 

-104 

-005 

203 

-157 

072 

106 

110 

246 

-H07 

C105 

-004 

-035 

-029 

-102 

065 

024 

-061 

078 

014 

148 

072 

318 

C106 

-077 

-010 

-145 

078 

-055 

-023 

-055 

085 

-096 

273 

-096 

062 

C107 

092 

038 

-178 

-110 

005 

-030 

-099 

-030 

083 

254 

-013 

-037 

C108 

-019 

-021 

-051 

085 

-135 

045 

-202 

102 

085 

109 

085 

027 

CI  09 

200 

050 

-072 

007 

-030 

162 

114 

100 

007 

263 

007 

083 

Clio 

-028 

-035 

-099 

051 

-158 

-102 

-094 

065 

-008 

229 

111 

153 

cm 

153 

090 

-047 

134 

094 

-037 

-065 

101 

-013 

185 

-013 

086 

C112 

047 

106 

106 

-131 

-106 

-031 

-106 

-092 

261 

080 

000 

074 

C113 

100 

033 

-079 

-126 

075 

269 

-020 

106 

135 

093 

-039 

094 

C114 

203 

105 

-091 

038 

065 

094 

-027 

015 

038 

012 

-047 

005 

C115 

024 

-025 

-039 

083 

-099 

061 

110 

151 

-110 

254 

-013 

314 

C116 

050 

-086 

-129 

-079 

-072 

-087 

-072 

033 

-079 

069 

-079 

142 

C118 

161 

-088 

-039 

180 

005 

061 

110 

151 

-110 

077 

083 

-037 

C119 

026 

023 

-054 

-086 

-078 

128 

051 

017 

033 

027 

-086 

226 

C401 

158 

109 

100 

-021 

091 

009 

165 

009 

115 

101 

047 

022 

C402 

024 

006 

-010 

-013 

-116 

-104 

023 

015 

-077 

155 

115 

058 

C403 

050 

080 

056 

-079 

-072 

152 

067 

1  CO 

152 

048 

070 

048 

142 

C405 

216 

085 

282 

130 

086 

008 

148 

168 

-098 

187 

073 

031 

C406 

292 

197 

150 

-092 

-070 

091 

243 

023 

-092 

-105 

052 

-055 

C407 

120 

105 

Old 

-020 

024 

-059 

024 

051 

-138 

082 

039 

137 

C408 

(718) 

050 

214 

-059 

042 

038 

186 

-023 

-059 

142 

139 

-037 

C409 

(686) 

254 

-185 

084 

259 

214 

089 

116 

-086 

-004 

105 

C413 

(732) 

014 

122 

109 

267 

109 

-120 

001 

-053 

-030 

C414 

(757) 

198 

-039 

097 

048 

-022 

158 

071 

-047 

C415 

(741) 

075 

125 

169 

-104 

-Oil 

-104 

065 

C416 

* 

i 

(831) 

169 

106 

048 

-066 

048 

015 

C417 

(778) 

169 

-104 

-072 

-104 

-027 

C418 

(724) 

-039 

040 

-126 

094 

C420 

(827) 

-068 

-115 

-047 

C421 

(645) 

158 

167 

C603 

(633) 

207 

C604 

(787) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlations  Among  Package  02  Competencies 


C605     C606.  C607 


COMPETENCY 
C608     0609     C610  C611 


C613     C614     C615     C616  0617 


076 
148 
061 
206 
175 

-082 
076 

-047 
178 

-154 
010 

-123 
052 
045 
076 

-050 
076 

-054 
114 
160 

-050 
017 

-103 

-072 
.079 
058 

-016 

-072 

-065 
052 

-065 

-079 
067 
035 
067 

045 


-062 
051 
141 
172 
028 

-GDI 

-062 
070 
115 
054 
020 

-124 
075 

-001 
203 
123 
004 
Oil 

-096 

-189 
035 
142 
006 

-171 

-022 
131 
128 
051 

-116 
135 
023 

-044 
051 
155 
051 
116 


037 
-051 

122 

163 
-049 
-190 

037 
-197 
-117 
-055 
-122 
-215 
-020 
-092 
-090 

096 
-090 
-092 

003 
-110 
-157 
-044 
008 
-040 
-030 
071 
-017 
017 
060 
-020 
-140 
-191 
-044 
-023 
139 
130 


-056 
128 
058 
128 
148 
025 
-056 
155 
-151 
219 
-056 
031 
010 
-023 
066 
099 
005 
-030 
101 
-021 
018 
-033 
-130 
-016 
-068 
-041 
085 
026 
-082 
-154 
-209 
010 
-032 
080 
202 
189 


-037 
042 
002 
120 
-050 
010 
-037 
-085 
023 
-009 
-048 
134 
015 
-072 
-125 
-090 
-037 
-098 
022 
058 
026 
031 
-121 
-023 
-037 
050 
-030 
038 
-118 
-064 
-027 
015 
207 
115 
038 
-149 


-026 
114 
044 
117 
142 
193 

-090 
006 
188 
258 
120 
000 
095 
019 
228 
012 
101 
143 
048 

-068 

-073 
181 
151 

-040 
144 
071 
160 
078 

-007 
152 
060 
037 
017 
126 
139 
019 


086 
016 
083 
161 
097 
-176 
-069 
-101 
-151 
-139 
-037 
-105 
041 
-108 
-070 
-004 
-070 
-020 
026 
-173 
-107 
-010 
-104 
-059 
-151 
075 
-035 
070 
022 
251 
-140 
-169 
220 
-016 
295 
300 


013 
306 
209 
167 
228 
241 
075 
195 
184 
199 
083 
084 
232 
045 
260 
240 
137 
197 
042 
013 
158 
095 
019 
-018 
-028 
043 
-013 
-068 
-029 
121 
-094 
177 
-008 
120 
051 
261 


115 
056 
057 
045 
097 
-007 
-079 
031 
-098 
-029 
-057 
022 
052 
-022 
-014 
-065 
050 
-004 
255 
039 
-065 
036 
-022 
-012 
035 
127 
004 
030 
072 
-006 
072 
052 
21  > 
205 
092 
-022 


-004 
-085 
056 
161 
235 
113 
055 
215 
103 
070 
-026 
-040 
068 
037 
-004 
132 
055 
-063 
-110 
035 
-024 
-030 
-050 
-078 
-018 
036 
045 
079 
092 
-039 
-155 
174 
-035 
240 
135 
192 


-044 
080 
020 
104 
033 
-Oil 
041 
118 
066 
078 
072 
231 
236 
069 
-044 
019 
-044 
004 
-025 
-016 
132 
052 
-066 
-041 
125 
013 
016 
-054 
-034 
-071 
-123 
005 
110 
036 
110 
-005 


051 
042 
166 
057 
265 
-043 
-125 
-029 
023 
-009 
-048 
-104 
094 
-072 
-037 
026 
226 
010 
-039 
174 
-090 
134 
-055 
-130 
-037 
-005 
-091 
038 
065 
015 
-118 
015 
122 
012 
207 
158 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communallty  In  the  factor  analysis, 
Leading  decimal  points  are  omitted. 
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E.28 


TABLE  E6 


Correlations  Among  Package  02  Competencies 


COMPE- 
TENCY 

C620 

C622 

C625 

C626 

COMPETENCY 
C627  C628 

C629 

C630 

C902 

C903 

C90'l 

C905 

ClOl 

-026 

-129 

203. 

115 

061 

013 

-134 

-004 

044 

05.7 

-027 

052 

CI  02 

031 

025 

051 

-099 

001 

-015  . 

009 

122 

047 

-061 

134 

139 

C103 

044 

126 

017 

037 

096 

248 

192 

168 

005 

084 

240 

262 

C104 

117 

043 

269 

-031 

137 

032 

089 

053 

292 

073 

250 

121 

C105 

-049 

136 

-012 

013 

024 

ICO 

153 

113 

159 

186 

-110 

175 

-015 

CI  06 

001 

143 

039 

070 

-023 

055 

170 

-016 

056 

-106 

-073 

037 

C107 

-026 

127 

004 

-051 

061 

199 

199 

096 

-077 

057 

117 

-184 

CI  08 

006 

282 

155 

-007 

102 

195 

046 

-085 

-022 

-069 

021 

-Oil 

CI  09 

014 

300 

024 

051 

038 

-028 

051 

161 

006 

050 

194 

160 

Clio 

062 

131 

095 

058 

-046 

122 

161 

013 

092 

-006 

017 

-059 

cm 

-025 

137 

020 

058 

101 

224 

185 

153 

039 

193 

135 

010 

C112 

172 

000 

056 

-127 

-031 

000 

099 

017 

072 

045 

036 

060 

C113 

-020 

005 

135 

-004 

188 

177 

221 

061 

085 

033 

321 

-080 

C114 

-092 

144 

116 

058 

094 

-009 

131 

051 

010 

026 

048 

-051 

C115 

037 

212 

-129 

115 

061 

-049 

282 

-105 

-016 

-076 

117 

-007 

C116 

-073 

019 

035 

123 

152 

240 

233 

057 

-026 

-055 

-028 

-140 

C118 

-090 

297 

-129 

115 

-030 

013 

115 

-105 

105 

-076 

117 

052 

C119 

-013 

109 

Oil 

-003 

017 

121 

-003 

042 

021 

104 

131 

054 

C401 

003 

-085 

044 

-070 

nno 

-070 

033 

046 

-061 

081 

-037 

C402 

016 

-016 

-101 

025 

-044 

-110 

025 

-062 

119 

123 

-080 

-017 

C403 

012 

019 

-053 

013 

-087 

-088 

013 

-076 

134 

297 

162 

094 

C405 

144 

-099 

142 

023 

168 

-052 

-075 

-069 

-008 

050 

126 

200 

0406 

056 

-130 

006 

048 

023 

-073 

-014 

-157 

-069 

086 

-040 

210 

C407 

-001, 

-145 

-009 

-109 

-004 

-018 

-160 

001 

173 

148 

-090 

052 

C408 

144 

066 

-068 

051 

100 

057 

-006 

-114 

047 

141 

047 

-042 

C409 

'031 

-094 

048 

-022 

089 

-035 

-074 

-025 

103 

163 

098 

-048 

C413 

027 

-161 

-01 0 

118 

046 

-142 

-055 

-039 

106 

056 

009 

222 

C414 

078 

110 

243 

099 

-039 

051 

-061 

083 

020 

-079 

166 

023 

C415 

-007 

-034 

092 

133 

-020 

-029 

-127 

005 

134 

-072 

-012 

021 

C416 

-134 

005 

-044 

071 

025 

065 

-079 

061 

031 

152 

321 

-080 

C417 

-073 

-034 

-116 

-040 

-114 

-223 

-127 

-099 

-055 

067 

-086 

-040 

C418 

-077 

236 

-044 

-004 

025 

010 

071 

061 

085 

033 

128 

-027 

C420 

-044 

110 

179 

-061 

048 

051 

019 

373 

-096 

-079 

166 

-033 

C421 

163 

186 

116 

010 

199 

084 

205 

018 

096 

-008 

171 

106 

C603 

200 

-054 

243 

099 

135 

-008 

-141 

-110 

020 

176 

028 

023 

C604 

019 

-005 

058 

-087 

094 

-063 

-014 

-037 

010 

142 

048 

001 

NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  comnunality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlations  Among  Package  02  Competencies 


NOTE; 


COMPETENCY 


TENCY 

C906 

C910 

C911 

091 2 

C915 

C917 

COOl 

C002 

C003 

C004 

C009 

COlO 

CI  01 

OS  7 

1  ^5 

CO'J 

0^1 
UO  1 

n  7Q 

nco 
-Uoo 

Uoo 

300 

1  ^1 

171 

-165 

-018 

CI  02 

uou  ■ 

"UOl 

HAP 
-Uoo 

- 1  1 0 

Uoo 

-U/4 

013 

-123 

Ol  o 

019 

150 

079 

C103 

001 

104 

-Uoo 

uou 

ni  ^ 
-U  1  0 

91  Q 

c  1  o 

r/i  o 
U4o 

1 

lOo 

nil  A 

-049 

1  o  ^ 

105 

273 

1  76 

C104 

-02^ 

-OdiT 

— U4U 

uou 

nQ9 

-UUl 

Goo 

-Oil 

o  oo 

083 

002 

188 

C105 

-18Q 

01 7 

1  oy 

- 1  oy 

1  09 

-U/U 

-046 

111 

-111 

O  1  ^ 

-016 

108 

099 

CI  06 

021 

-tU  1 

1  Q9 

- 1  oc 

-  loo 

-Uo/ 

rt  71 

-0/1 

1  oo 

-138 

1  il  n 

-148 

205 

010 

C107 

IQO 

042 

1  O  1 

nci 
Uo  1 

-U  1 4 

-1  1  / 

007 

ooo 

-020 

O  1  /\ 

-010 

125 

158 

no8 

01 R 

-008 

U  1  U 

090 
-UtU 

071 

-u  / 1 

0  79 

U/0 

-054 

ooo 

-023 

1  o  ^ 

-127 

128 

154 

C109 

050 

-058 

-077 

— UoO 

09Q 

-u^o 

-uuy 

91  1 
1  1 

1  oo 

1  m 

-107 

o  r  ^ 

057 

209 

100 

Clio 

07fi 

-u  t  ^ 

1  09 
-  1  Ut 

090 

U40 

-043 

ooo 

-082 

1 

-156 

167 

243 

cm 

-110 

-025 

— uu  / 

101 
1  U  1 

0Q9 
Uo^ 

1  /in 
I4U 

U^/ 

Ol  A 

214 

1 1  ii 
-114 

1  O/" 

186 

o  r  ^ 

257 

083 

C112 

1^5 

-146 

1  AO 
1  HU 

OQl 
-UO  1 

ICO 
1  00 

ni  n 
Ul  U 

lie 

1 1o 

054 

o  oo 

000 

049 

199 

CI  13 

-087 

128 

UO  1 

-09^ 

-u^o 

-uoo 

lli  Q 

loo 

-Uoo 

11/1 

114 

1  oo 

102 

1  il  ^ 

147 

007 

225 

C114 

026 

118 

uoo 

01  R 
U  1  0 

uy  1 

-uoy 

-Olb 

1  oo 

138 

1  oo 

122 

057 

039 

C115 

-076 

-082 

-Olfi 

U  1  u 

-045 

— UHO 

-1  90 

^AA 

-  1  *fH 

1  7C 
- 1  /D 

-Ubb 

1  >l  o 

-149 

oco 

050 

O  ^1 

271 

099 

C116 

121 

104 

1  Uu 

-.041 

-Uo  / 

1  09 

1  y^ 

n9n 

mo 

-Uo9 

o  c  o 

058 

O^  1 

361 

164 

C118 

-076 

-082 

1  65 

-.045 

-1  90 

1  AA 

-Uoo 

UU/ 

o  oc 

-Ooo 

Ol  o 

-010 

1  o  r 

125 

-077 

CI  19 

088 
uoo 

-209 

-1 . 
-  1  HO 

OQA 

1  cc 
1  DO 

-Uoo 

191 

o  oo 

-088 

1  oo 

100 

139 

045 

C401 

126 

04Q 

0fi9 
UO^ 

079 

n9Q 
Uco 

UUo 

o1 1 

1  TO 

1 73 

oo^ 

-006 

-134 

-019 

C402 

-141 

Oil 

1  5Q 

-099 

-U*f*f 

1  09 

- 1  oc 

U40 

-^02 

OOT 

027 

1^1 
-151 

-020 

-116 

C403 

121 

104 

054 

050 

152 

021 

-098 

-010 

186 

-102 

07^? 

uoo 

C405 

-028 

-067 

206 

175 

062 

-002 

094 

001 

015 

191 

-234 

-048 

C406 

-014 

154 

066 

087 

-044 

-022 

091 

194 

-174 

211 

024 

-115 

C407 

-176 

-040 

210 

162 

106 

067 

079 

193 

-065 

224 

014 

-010 

C408 

-041 

026 

088 

106 

038 

124 

073 

215 

068 

223 

010 

060 

C409 

-003 

023 

254 

179 

089 

127 

024 

209 

179 

356 

039 

233 

C413 

056 

032 

273 

118 

046 

-086 

096 

062 

171 

158 

-080 

203 

C414 

176 

-086 

-038 

-125 

-039 

-094 

084 

-074 

085 

-149 

036 

-102 

C415 

067 

051 

134 

257 

-020 

005 

-212 

-034 

200 

203 

-081 

Oil 

C416 

-087 

128 

085 

038 

025 

168 

-032 

114 

160 

201 

-124 

172 

C417 

067 

180 

134 

042 

169 

005 

-150 

-034 

066 

076 

-005 

-112 

C418 

033 

128 

194 

100 

106 

-122 

-191 

-052 

-013 

038 

137 

-040 

C420 

176 

033 

020 

073 

135 

154 

084 

104 

270 

-091 

-034 

182 

C421 

070 

027 

-045 

-019 

146 

-022 

084 

101 

-041 

009 

195 

070 

C603 

-079 

-086 

-038 

007 

-039 

030 

084 

104 

-038 

026 

036 

-045 

C604 

-090 

010 

-043 

023 

-143 

034 

-059 

038 

-197 

069 

120 

090 

Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 
correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlations  Among  Package  02  Competencies 


COMPETENCY 


TENCY 

COl  1 

\JJ\d 

COl  3 

0014 

0015 

001 7 

001 8 

0019 

CI  01 

095 

-068 

072 

077 

061 

-051 

031 

-004 

CI  02 

134 

252 

079 

036 

080 

121 

150 

187 

CI  03 

-129 

-010 

079 

043 

263 

158 

047 

106 

Clb4 

-169 

-125 

021 

068 

127 

000 

021 

053 

CI  05 

-073 

052 

-065 

148 

142 

020 

012 

159 

CI  06 

-066 

023 

-002 

-116 

091 

-051 

-020 

044 

C107 

-064 

064 

-115 

018 

179 

017 

-178 

-105 

CI  08 

-088 

-028 

-055 

-041 

113 

Oil 

-095 

-020 

CI  09 

131 

103 

-009 

-060 

113 

047 

-072 

024 

Clio 

-076 

080 

-143 

048 

046 

042 

073 

075 

cm 

094 

161 

188 

141 

251 

039 

-100 

001 

CI  12 

-148 

000 

-016 

000 

050 

-Oil 

200 

085 

C113 

-021 

008 

-048 

040 

180 

153 

109 

061 

CI  14 

177 

050 

026 

115 

147 

074 

-030 

051 

CI  15 

175 

261 

-022 

-099 

-058 

-051 

-039 

-105 

CI  16 

002 

291 

040 

226 

214 

045 

-129 

057 

C118 

095 

261 

-022 

-099 

-058 

-051 

100 

-004 

C119 

-112 

168 

140 

-045 

087 

105 

117 

165 

0401 

014 

054 

125 

019 

-119 

024 

051 

033 

C402 

-057 

-111 

032 

-079 

-111 

Oil 

082 

071 

C403 

002 

-144 

040 

-008 

058 

045 

148 

-076 

0405 

-124 

-114 

150 

048 

094 

110 

036 

050 

0406 

024 

046 

107 

071 

135 

076 

-058 

069 

0407 

006 

057 

301 

189 

151 

062 

179 

062 

0408 

131 

148 

183 

142 

113 

326 

023 

'  230 

C409 

160 

020 

190 

208 

245 

021 

080 

025 

0413 

087 

-066 

192 

206 

096 

012 

083 

100 

0414 

079 

107 

059 

-182 

-086 

-196 

-053 

-110 

0415 

113 

017 

-012 

174 

097 

-106 

-024 

005 

0416 

051 

067 

120 

-066 

021 

-031 

-016 

-030 

0417 

030 

085 

183 

-134 

097 

-035 

-024 

-099 

0418 

-164 

-051 

036 

-066 

-032 

031 

046 

-120 

0420 

079 

-146 

-121 

-068 

027 

000 

081 

-110 

0421 

-085 

095 

124 

068 

187 

159 

042 

018 

0603 

003 

044 

-031 

045 

141 

131 

014 

083 

0604 

-032 

107 

-055 

012 

044 

136 

092 

138 

NOTE:    Correlations  are  ph1  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis* 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlation-  Among  Package  02  Competencies 


COMPE- 
TENCY 


COMPETENCY 

C605     C606     C607     C608     C609     C610     C611     C613     C614     C615     C616  C617 


C605 

C606 

C607 

CiOB 

C609 

C610 

C611 

C613 

C614 

C615 

C616 

C617 

C620 

C622 

C625 

C626 

C627 

C628 

C629 

C630 

C902 

C903 

C904 

C9'05 

C906 

C910 

C911 

C912 

C915 

C917 

COOl 

C002 

C003 

C004 

C009 

COlO 


(821)     160       225  075 
(742)     142  302 
(685)  208 
(707) 


-082 
116 

-092 
030 

(731) 


133 
395 
074 
206 
130 
(839) 


128 
240 
335 
256 
232 
138 
(865) 


293 
300 
101 
144 
207 
258 
101 
(751) 


050 
125 
018 
259 
316 
264 
157 
131 
(785) 


021 
113 
193 
170 
037 
193 
090 
253 
238 
(782) 


-085 
097 

-106 
152 
218 
110 
016 
078 
292 
159 

(627) 


171 
174 
241 
189 
081 
241 
368 
163 
203 
192 
069 
(792) 


NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlations  Among  Package  02  Conpe.tencles 


COMPETENCY 


can 

0622 

vQc.O 

CJi9Q 

CQ02 

C903 

C904 

C905 

-051 

038 

064 

-088 

-079 

025 

152 

221 

005 

-050 

095 

-113 

227 

267 

252 

080 

314 

095 

246 

004 

119 

-053 

299 

139 

-047 

002 

058 

-015 

-020 

101 

091 

101 

001 

012 

102 

-028 

167 

100 

181 

015 

230 

144 

066 

005 

171 

-143 

012 

180 

186 

069 

116 

-087 

094 

-009 

131 

138 

168 

-090 

110 

001 

235 

272 

227 

196 

152 

023 

249 

-154 

154 

-073 

Oil 

196 

-010 

016 

240 

134- 

181 

053 

069 

086 

-082 

-004 

094 

-014 

062 

131 

136 

007 

121 

084 

212 

137 

204 

-088 

193 

014 

223 

127 

210 

001 

168 

-029 

108 

-014 

265 

-065 

031 

082 

193 

159 

270 

128 

281 

034 

177 

-004 

220 

-103 

220 

007 

272 

058 

097 

046 

159 

131 

117 

-044 

092 

132 

094 

078 

-147 

218 

174 

-014 

015 

045 

058 

-037 

-043 

026 

110 

155 

(756) 

110 

227 

091 

381 

062 

038 

154 

040 

012 

192 

046 

(870) 

041 

-024 

-071 

183 

258 

-044 

195 

-093 

155 

028 

(761) 

080 

314 

054 

080 

004 

039 

035 

062 

139 

(644) 

221 

007 

035 

-051 

020 

013 

079 

200 

(813) 

121 

221 

061 

031 

033 

256 

132 

(675) 

161 

137 

-019 

-088 

237 

-168 

(846) 

-051 

020 

-097 

198 

-045 

(765) 

105 

057 

188 

-066 

(739) 

054 

056 

073 

(728) 

067 

094 

(815) 

-027 

(806) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E6 

Correlations  Among  Package  02  Competencies 


COMPETENCY 


C906  C910 


roi  ^ 

PQl  7 

LOOl 

cooz 

C003 

C004 

CO  09 

ncn 
UDU 

1  AC 

-157 

-047 

009 

-108 

\J/0 

fiAC 

r>  o  O 

083 

070 

210 

-068 

1  on 

\J\iZ 

050 

087 

244 

-159 

U  1  u 

AOO 

082 

038 

-025 

-049 

n7Q 

AOO 

038 

1  Ail 

194 

-037 

-133 

092 

Al  1 

Oil 

046 

205 

-021 

-noQ 

in7 
lU/ 

144 

1  il  A 

140 

147 

067 

-120 

"UHO 

AAC 

-005 

-121 

180 

032 

1  72 

1  1  o 

118 

O  O  T 

237 

043 

-095 

UOo 

nc7 

1 1  o 

AC  A 

-059 

Al  O 

-012 

133 

118 

1  97 

-Uoo 

Al  O 

018 

1 1  n 

118 

AAA 

099 

-143 

"UvH 

001 

ATA 

-0/0 

027 

069 

-070 

uo/ 

U04 

AOA 

089 

006 

013 

-067 

O  1  £ 

"Uoo 

ni  £ 
UlD 

1  on 
-139 

064 

A  Ail 
-004 

104 

.noA 

•UUH 

U40 

1  OA 

1^4 

1  QT 

197 

AAA 

090 

-020 

-n7Q 

UOo 

UUU 

1  CO 

152 

078 

AAA 

•  089 

106 

168 

074 

335 

160 

256 

072 

-046 

290 

192 

184 

076 

143 

-012 

071 

-052 

-oil 

000 

046 

128 

089 

-030 

115 

-117 

068 

108 

-01  n 

248 

-046 

091 

041 

170 

180 

117 

152 

277 

-020 

234 

101 

138 

-023 

-001 

077 

018 

226 

021 

152 

-061 

132 

-031 

023 

-003 

-049 

-088 

121 

033 

-065 

-098 

072 

101 

-183 

073 

240 

234 

014 

040 

-088 

-049 

049 

-078 

148 

-016 

-071 

248 

216 

030 

162 

035 

-130 

047 

244 

389 

-089 

(818) 

052 

074 

114 

160 

038 

007 

(719) 

210 

237 

154 

160 

045 

(793) 

244 

075 

004 

-087 

(691) 

182 
(708) 

202 
183 
(820) 

112 
-104 
-005 
(730) 

COlO 


-050 
035 
096 
018 
142 
-073 
099 
158 
106 
132 
019 
026 
012 
019 
299 
-097 
-087 
-088 
013 
190 
-106 
-055 
067 
-062 
(730) 


124 
Oil 
065 
-105 
-098 
-013 
-020 
-032 
075 
083 
-101 
118 
-170 
-101 
-153 
-105 
017 
-032 
202 
042 
021 
104 
043 
-019 
-059 
(783) 


005 
-001 
001 
-085 
-043 
-037 
-036 
018 
-046 
113 
041 
-096 
040 
-062 
-081 
070 
085 
055 
070 
-016 
092 
054 
-030 
-069 
-026 
021 
(784) 


079 

024 

057  * 
-068 

083 
-030 

009 

184 

035 

103 

125 
-037 

057 

066 

069 

051 

100 

057 

165 
-045 

047 

141 
-051 
-082 

050 
-058 

502 
(754) 


-035 
352 
098 
349 
142 
284 
198 
061 
060 
073 
214 
039 
172 
164 
079 

-010 
278 
170 
234 
041 
045 

-070 
249 

-002 
008 

-100 
116 
100 
013 
250 
090 
188 
229 
205 
103 

(780) 


NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.34 


TABLE  E6 


Correlations  Among  Package  02  Competencies 


COMPE- 

COMPETENCY 

TENCY 

con 

C012 

C013 

C014 

C015 

C017 

C018 

C019 

-036 

-075 

-050 

UwU 

uu  1 

-117 

076 

w  /  w 

049 

020 

WkW 

032 

W  Wk 

116 

1  1  w 

-033 

WWW 

056 

w«/U 

036 

004 

C607 

056 

www 

128 

-120 

126 

1  bW 

-021 

— Ub  1 

043 

-017 

-026 

-046 

w^w 

-006 

www 

-010 

w  Iw 

152 

1  Wb 

-059 

W  w^ 

113 

1  1  w 

085 

066 

w 

050 

www 

107 

1  w/ 

012 

W  1  b 

096 

074 

335 

-037 

C610 

WW  1  W 

156 

1  WW 

003 

www 

-002 

WWb 

163 

017 

043 

V^w 

160 

164 

cell 

wU  1  1 

106 

1  w  w 

025 

WbW 

055 

www 

-016 

006 

000 

W  WW 

019 

008 

C613 

wU  1  w 

022 

w  mmmm 

120 

1  bW 

-028 

WbU 

-025 

119 

117 

-084 

158 

013 

C614 

WW  1  ~ 

-131 

1  W  1 

196 

1  W 

071 

w  *  1 

167 

058 

022 

W  bb 

228 

-079 

C615 

WW  ■  */ 

-116 

1  1  w 

013 

w  1  w 

-064 

ww» 

066 

182 

1  Ob 

-080 

045 

-063 

C616 

wV  1 V 

-042 

-079 

w  /  ^ 

-142 

1  Tb 

136 

1  WW 

116 

1  lU 

000 

w  wU 

135 

-044 

C617 

wU  1  / 

-032 

U  Wb 

-008 

wwU 

137 

1  w» 

064 

w  w~ 

044 

015 

V  1  w 

-030 

W  WW 

226 

C620 

-045 

086 

175 

1  #  w 

089 

W 

OQP 

086 

WWW 

?04 

101 

C622 

WWbb 

026 

W  bW 

033 

www 

-142 

IrTb 

036 

W  WW 

-135 

000 

w  W 

075 

041 

C625 

-109 

-111 

III 

C/32 

\r  Wb 

038 

w  WV 

085 

-034 

w  w^ 

-148 

137 

C626 

wwbU 

146 

017 

V  1  r 

085 

www 

059 

W  War 

086 

042 

W*tb 

002 

032 

ri527 

-021 

067 

288 

bOO 

093 

V7  w 

127 
lb/ 

oil 

046 

w^w 

-120 

■  b  W 

-125 

-001 

W  1 

087 

120 

1  bW 

uoo 

042 

-013 

w  ■  w 

013 

w  ■  w 

-118 

126 

1 

-147 

-087 

wU/ 

-Oil 

— U  1  1 

-183 

—  1  Ow 

002 

W  Wb 

-051 

w  w  ■ 

016 

064 

072 
u  /b 

018 

w  1  O 

002 

017 

031 

WW  ■ 

-004 

-066 

06^ 

222 

bbb 

309 

nc;i 

231 

b  W  ■ 

105 

C903 

-103 

-057 

287 

070 

136 

225 

-037 

057 

UoU 

-  ICO 

102 

036 

108 

-098 

wJO 

C905 

-086 

004 

052 

141 

-on 

100 

099 

170 

C906 

002 

030 

-083 

-086 

058 

-135 

-037 

057 

C910 

-n2 

087 

140 

027 

087 

105 

-140 

165 

C911 

173 

102 

166 

132 

091 

-010 

106 

-016 

C912 

077 

-033 

055 

182 

194 

-093 

071 

024 

C915 

-092 

-110 

120 

199 

127 

-031 

172 

-120 

C917 

-080 

153 

312 

167 

172 

284 

-086 

179 

C001 

-062 

024 

216 

153 

270 

130 

178 

061 

C002 

-035 

109 

366 

065 

208 

115 

105 

068 

C003 

115 

-070 

124 

146 

037 

Oil 

038 

-149 

C004 

038 

114 

285 

400 

218 

082 

074 

-010 

C009 

061 

209 

156 

153 

212 

049 

-080 

125 

C010 

132 

021 

-CI  5 

243 

193 

120 

121 

099 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  comrnmallty  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.35 
TABLE  E6 


Correlations  Among  Package  02  Competencies 


COMPE- 
TENCY 

con 

0012 

C013 

C014 

COMPETENCY 
C015  C017 

C018 

C019 

con 

(760) 

092 

-010 

148 

172 

013 

032 

-064 

C012 

(726) 

148 

133 

101 

089 

071 

129 

C013 

(807) 

179 

161 

no 

063 

258 

C014 

(769) 

222 

239 

083 

254 

C015 

(789) 

170 

014 

002 

C017 

(641) 

012 

289 

Cd18 

(671) 

-039 

C019 

(843) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.36 
TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE- 
TENCY 


COMPETENCY 


C363 

C364 

C366 

C368 

C371 

C372 

C373 

C374 

C375 

016 

-086 

152 

069 

-101 

-054 

062 

-032 

088  - 

072 

019 

019 

137 

-010 

277 

048 

160 

074 

250 

261 

065 

039 

000 

171 

279 

293 

188 

(843) 

058 

187 

131 

049 

262 

197 

566 

181  . 

(804) 

-022 

061 

-054 

067 

219 

111 

-086 

(811) 

134 

275 

067 

113 

111 

271 

(740) 

202 

231 

031 

130 

-025 

(854) 

038 
(843) 

-023 
097 

025 
203 

004  • 
124 

(813) 

264 

332 

(826) 

-032  • 
(815) 

C310 

C312 

C361 

C363 

C364 

C366 

C368 

C371 

C372 

C373 

C374 

C375 

C376 

C378 

C401 

C402 

C4G4 

C405 

C406 

C407 

C4G8 

C409 

C411 

C412 

C413 

C414 

C415 

C418 

C421 

C501 

C502 

C506 

C507 

C508 

C541 

C542 


(770)     360  021 
(852)  236 
(813) 


NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communallty  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


E.37 


TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE- 


COMPETENCY 


TENCY 

C376 

C378 

C401 

C402 

C404 

0405 

C40S 

C407 

C408 

C409 

C411 

C412 

C310 

188 

088 

Ail  A 

049 

256 

-094 

006 

AC  O 

06Z 

A  OP 

035 

0Z6 

1  AA 

100 

1  OC 

-136 

AC  O 

06  Z 

C3IZ 

314 

-069 

146 

066 

-101 

1  Ail 

104 

1  oo 

1Z9 

1  il  o 

149 

ooc 

085 

Oil  c 

045 

oi  ? 

017 

Oil  o 

048 

tool 

Zio 

105 

Zo3 

001 

Z81 

nno 
09Z 

110 

1Z7 

oco 

ObZ 

Oil  t 

-047 

001 

OZl 

oi  c 

Z15 

occ 

056 

tJDO 

341 

01 D 

ZZ3 

OT  T 
311 

144 

087 

016 

Ol  o 

ZIO 

-Oil 

oo? 

-007 

ooc 

Z96 

1  oo 

1Z3 

C3b4 

-Odd 

-086 

1  oo 

183 

T  T  C 
115 

-IZo 

-041 

oco 

ObZ 

039 

oco 

-059 

-185 

0  01 

^  OZl 

ooo 

-099... 

C366 

39Z 

-086 

noi 

-OZl 

TIC 

115 

o  on 

-039 

-041 

O  CO 

ObZ 

1  on 

-138 

-125 

TIC 

116 

Oil? 

-047 

A  A  ? 

007 

C368 

14Z 

163 

n  oo 

038 

070 

-037 

056 

001 

m  n 

018 

AP^ 

-056 

-006 

O  OA 

230 

AOl 

031 

C371 

173 

004 

o  oc 

035 

-016 

O  OC 

Z3d 

^  AO 

10Z 

125 

Ail  1 

-041 

Apn 

-050 

Of  c 

066 

-035 

A  ?1 

071 

-UDH 

1  IH 

UOh 

CCC 

977 
CI  1 

UDo 

nQ7 

C373 

130 

-074 

156 

262 

-009 

062 

-058 

050 

-013 

-004 

153 

398 

C374 

168 

-032 

085 

218 

010 

021 

065 

057 

-028 

-035 

262 

061 

C375 

272 

-064 

136 

338 

017 

224 

-030 

-040 

-058 

023 

037 

062 

C376 

(840) 

105 

215 

326 

031 

030 

076 

-021 

-047 

148 

072 

-019 

C378 

(729) 

136 

-071 

-094 

006 

-123 

-040 

026 

-054 

037 

-074 

C401 

(775) 

138 

199 

190 

207 

147 

158 

109 

041 

156 

C402 

(837) 

015 

142 

-044 

031 

024 

006 

143 

116 

C404 

(678) 

168 

293 

216 

038 

259 

118 

288 

C405 

(793) 

207 

342 

216 

085 

101 

192 

C406 

(837) 

185 

292 

197 

267 

188 

C407 

(732) 

120 

105 

197 

184 

C408 

(710) 

050 

083 

-013 

C409 

(699) 

023 

202 

C411 

(822) 

308 

C412 

(818) 

i 
% 
\ 
I 
I 
I 
I 
I 

[erIc 


C413 
C414 
C415 
C418 
C421 
C501 
C502 
C506 
C507 
C508 
C541 
C542 


NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  comrnmallty  in  the  factor  analysis. 
Leadinc?  >A«c1mal  points  are  omitted. 
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E.38 


TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPETENCY 


C413  C414 


-054 
-069 
-035 
-046 
-187 
081 
-047 
016 
-016 
-083 
-099 
032 
012 
032 
100 
-010 
172 
282 
150 
010 
214 
254 
095 
069 
(769) 


-086 
131 
131 
-071 
071 
-115 
-013 
-054 
-072 
-099 
-008 
-086 
065 
033 
-021 
-013 
-039 
130 
-092 
-020 
-059 
-185 
-115 
-099 
014 
(818) 


-078 
-084 
035 
-111 
-104 
-003 
-065 
-034 
-065 
026 
-094 
-078 
035 
051 
091 
-116 
169 
086 
-070 
024 
042 
084 
056 
026 
122 
198 
(787) 


Till  Q 

C4Z1 

C501 

C502 

C506 

017 

172 

-090 

004 

-059 

004 

127 

119 

-010 

090 

214 

000 

110 

231 

-045 

023 

098 

040 

163 

040 

-039 

102 

-031 

028 

048 

048 

102 

238 

110 

176 

032 

051 

401 

137 

-110 

-148 

-314 

333 

420 

245 

-079 

035 

194 

161 

142 

-009 

022 

101 

164 

-068 

-102 

-025 

145 

131 

-006 

128 

100 

025 

004 

-059 

-031 

199 

174 

173 

045 

240 

172 

025 

-101 

-059 

009 

101 

-071 

155 

-155 

015 

155 

094 

210 

-141 

025 

-119 

036 

159 

152 

168 

187 

-071 

102 

-028 

023 

-105 

037 

062 

-014 

051 

082 

074 

Oil 

-095 

-023 

142 

-073 

-050 

050 

089 

-086 

073 

119 

-003 

055 

-101 

137 

-035 

-032 

090 

022 

-OOi 

071 

078 

109 

001 

128 

-161 

-037 

048 

158 

-031 

-054 

-079 

169 

-Oil 

'■012 

-034 

-072 

(755) 

040 

-048 

-148 

033 

(774) 

-094 

-114 

-008 

(849) 

413 

164 

(841) 

132 

(794) 

C507 

C508 

C541 

C542 

-090 

u/o 

noo 

Oil 

1  OA 

noo 

-016 

107 

n7>i 
U/4 

165 

OQfi 

07  O 

Uoo 

-031 

no? 

238 

Ti? 
1 

970 
Civ 

Ice 

-025 

163 

1  fiO 

1  KO 
1  Oc 

-063 

-101 

1 OC 

01  o 

328 

1PA 

900 

-103 

1 A7 
10  / 

1  Id 
\  /O 

030 

www 

Oim 

1  CO 

1  oo 

-090 

w  ^w 

u/o 

no 

047 

10^ 

1  QO 

-090 

w^w 

u/o 

-071 

w  /  1 

1  % 

1 1Q 

OQil 

AAl 

-001 

036 

w  wV 

017 

uo^ 

039 

152 

074 

031 

037 

154 

075 

076 

017 

186 

027 

-077 

055 

026 

079 

263 

132 

100 

046 

-169 

071 

124 

108 

039 

101 

197 

096 

-016 

063 

032 

-041 

-091 

-031 

-086 

-032 

122 

-012 

051 

-063 

-118 

036 

128 

-006 

-143 

070 

172 

054 

115 

134 

254 

131 

026 

096 

109 

127 

-005 

411 

104 

106 

142 

(825) 

369 

192 

026 

(803) 

075 
(818) 

CIO 
260 
(750) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

I^rSif  iJl!5JS^*4*^r®  used  «  estimates  of  conmnallty  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.39 


TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE- 


COMPETENCY 


PAHA 

l<0  1  1 

C310 

-018 

-045 

-086 

271 

226 

092 

-030 

143 

172 

197 

-084 

083 

C312 

033 

073 

-093 

578 

402 

-on 

209 

038 

236 

089 

049 

109 

C361 

203 

093 

000 

131 

074 

-214 

-010 

-043 

-105 

-084 

-022 

080 

C363 

356 

240 

-136 

-007 

-052 

-179 

086 

-098 

-061 

027 

139 

018 

C364 

132 

150 

-022 

-022 

038 

-205 

-091 

-167 

-080 

051 

-032 

-206 

C366 

071 

150 

-022 

-022 

122 

-077 

085 

017 

070 

-127 

030 

135 

C368 

255 

103 

-013 

061 

152 

-031 

083 

120 

-037 

083 

140 

199 

C371 

-006 

314 

110 

-054 

-080 

-016 

100 

056 

082 

-132 

127 

-042 

C372 

310 

101 

067 

207 

045 

-128 

075 

041 

016 

-065 

330 

021 

C373 

-060 

m 

-099 

113 

-016 

-102 

022 

-055 

-133 

-075 

026 

002 

C374 

321 

194 

-008 

-008 

-009 

-192 

-082 

-212 

-091 

-030 

-069 

-148 

C375 

137 

030 

-086 

033 

118 

Oil 

195 

065 

-116 

121 

-004 

156 

C376 

246 

258 

000 

196 

074 

-034 

113 

043 

053 

000 

197 

080 

C378 

215 

105 

152 

-086 

010 

174 

120 

065 

-116 

273 

155 

083 

C401 

102 

156 

047 

047 

022 

-096 

101 

048 

026 

042 

255 

-110 

C402 

214 

021 

-013 

115 

058 

-189 

-021 

-068 

-173 

013 

039 

035 

C404 

-026 

044 

-039 

-126 

-143 

-044 

-099 

095 

041 

-102 

052 

-039 

C405 

010 

033 

073 

073 

031 

142 

-033 

181 

-010 

095 

036 

-030 

C406 

-048 

038 

-164 

052 

-055 

006 

-130 

151 

-104 

019 

-022 

-050 

C407 

023 

127 

-079 

039 

137 

-171 

-016 

-040 

-059 

-018 

-012 

-078 

C408 

089 

-016 

139 

139 

-037 

-022 

-068 

144 

-151 

-028 

035 

-018 

C409 

044 

-035 

116 

-004 

105 

131 

-041 

071 

075 

043 

127 

036 

C4n 

342 

144 

-115 

021 

101 

-091 

156 

-003 

-136 

132 

108 

-014 

C412 

-060 

-022 

-099 

113 

-016 

043 

-044 

014 

-047 

061 

167 

002 

C413 

-027 

-043 

081 

-053 

-030 

128 

085 

160 

-035 

-013 

004 

045 

C414 

-050 

-026 

-115 

071 

-047 

051 

026 

078 

070 

-068 

030 

079 

C415 

-078 

082 

097 

-104 

065 

-116 

-082 

-007 

022 

-029 

072 

092 

C418 

030 

044 

048 

-126 

094 

-044 

010 

037 

-169 

177 

052 

174 

C421 

032 

027 

_-012 

158 

167 

155 

080 

126 

-016 

120 

205 

239 

C501 

226 

180 

-031 

059 

026 

-153 

160 

-120 

-017 

-143 

192 

210 

C502 

048 

107 

028 

028 

-154 

-072 

-004 

-052 

082 

-238 

237 

009 

C506 

007 

063 

-079 

048 

-090 

123 

099 

-073 

099 

-088 

106 

-024 

C507 

167 

067 

059 

059 

-055 

032 

-010 

-002 

055 

-085 

131 

045 

C508 

215 

180 

-086 

152 

118 

on 

045 

-013 

076 

121 

234 

010 

C541 

372 

290 

154 

092 

-079 

-004 

-133 

059 

-144 

Oil 

168 

-029 

C542 

232 

183 

038 

122 

-072 

-117 

083 

-036 

-040 

-009 

091 

-066 

i 
I 
I 
I 
I 
I 
i 

I 


ERIC 


NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  Mhlch  Mere  used  as  estlnates  of  conmjnallty  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E7 

Correlations  Among  Package  03  Conpetencles 


COMPETENCY 

C616     C617    C618     C620     C622     C625     C626     C627    C628     C629  C630 


C905 


004 
-109 
-115 
-065 
-054 
110 
137 
058 
161 
071 
-132 
214 
000 
214 
-025 
-016 
-071 
052 
-066 
-041 
125 
013 
-035 
-023 
016 
-054 
-034 
005 
036 
096 
058 
019 
016 
004 
182 
144 


226 
199 
074 
-052 
-047 
038 
086 
144 
045 
080 
-063 
010 
074 
010 
-039 
174 
015 
134 
-055 
-130 
-037 
-005 
033 
176 
-091 
038 
065 
015 
012 
-055 
069 
142 
-137 
-098 
-079 
081 


091 
071 
-082 
073 
-067 
Oil 
-016 
036 
026 
-124 
-161 
-109 
137 
091 
173 
-097 
207 
140 
096 
272 
-075 
198 
-059 
054 
215 
-067 
123 
134 
-015 
074 
-033 
007 
150 
191 
-119 
-165 


-092 
-036 
-129 
072 
-167 
017 
120 
056 
316 
084 
-055 
143 
172 
065 
003 
016 
152 
144 
056 
-001 
144 
031 
086 
084 
027 
078 
-007 
-077 
163 
057 
110 
096 
175 
221 
182 
075 


109 
-010 
-115 
-122 
-136 
-054 
072 
-088 
-085 
-116 
-132 
-101 
000 
319 
-085 
-016 
-071 
-099 
-130 
-145 
066 
-094 
-035 
-116 
-161 
110 
-034 
236 
186 
-142 
-038 
019 
-142 
-101 
-093 
-005 


092 
143 
-079 
-001 
-141 
051 
273 
097 
064 
043 

-028 
092 
236 
092 
044 

-101 
015 
142 
006 

-009 

-068 
048 

-044 
043 

-010 
243 
092 

-044 
116 
156 
097 

-053 

-029 
174 
082 
116 


-003 
-017 
042 
-189 
-061 
019 
058 
188 
-088 
-029 
-198 
-003 
155 
100 
-070 
025 
071 
023 
048 
-109 
051 
-022 
-164 
-120 
118 
099 
133 
-004 
010 
085 
117 
123 
007 
-105 
-053 
-087 


-094 
004 
-092 
-038 
-126 
048 
169 
082 
182 
090 
-046 
128 
153 
017 
072 
-044 
025 
168 
023 
-004 
100 
089 
-009 
090 
046 
-039 
-020 
025 
199 
120 
159 
-087 
036 
128 
052 
094 


-184 
-126 

000 
-221 

111 
-187 

036 
-080 
-065 
-007 
-107 
-108 
-251 
-032 
-002 
-110 

010 
-052 
-073 
-018 
057 
-035 
-172 
-075 
-142 
051 
-02d 
010 
084 
-028 
078 
076 
-028 
-032 
-188 
099 


-003 
-113 
-070 
-133 
099 
019 
058 
-095 
-088 
-120 
-198 
-003 
-014 
202 
-070 
025 
-079 
-075 
-014 
-160 
-006 
-074 
-164 
-120 
-055 
-061 
-127 
071 
205 
007 
046 
013 
085 
-003 
001 
-087 


-082 
-088 
-051 
-124 
083 
-110 
-076 
-129 
-069 
016 
-173 
042 
-187 
-082 
033 
-062 
-120 
-069 
-157 
001 
-114 
-025 
-174 
-094 
-039 
083 
005 
061 
018 
-115 
-044 
-076 
-115 
-082 
-144 
138 


126 
019 
-060 
001 
-090 
194 
145 
278 
057 
-012 
-023 
126 
180 
054 
-037 
-017 
079 
200 
210 
052 
-042 
-048 
-004 
-012 
222 
023 
021 
-027 
106 
216 
178 
094 
-003 
054 
-031 
052 


MOTE: 


21[fi*'l'*®^^^S^®"*^-  .'"agonal  elements  are  squared  multiple 
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TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE- 
TENCY 


COMPETENCY 

C907     C908     C910     C911     C912     C914     C915  C916 


C917     COOl     C002  C003 


-101 
-109 
-058 
-065 
-054 
-054 
137 
-015 
038 
164 
-080 
109 
-058 
-101 
-025 
-016 
159 
-049 
125 
-145 
-109 
066 
-035 
164 
075 
-054 
-034 
-071 
-214 
096 
058 
019 
016 
109 
-093 
-080 


202 
080 
-127 
-133 
-141 
019 
058 
-024 
032 
153 
-044 
100 
042 
-003 
-070 
-030 
146 
023 
110 
-008 
-063 
1R5 
bi. 
244 
060 
-061 
-040 
-154 
-087 
-070 
-095 
-097 
-070 
-105 
-160 
-014 


-064 
-069 
105 
-067 
033 
033 
-119 
004 
-054 
-074 
-108 
-064 
-063 
-064 
049 
Oil 
128 
-067 
154 
-040 
026 
023 
-049 
197 
032 
-086 
051 
128 
027 
-090 
-101 
-059 
025 
088 
-084 
010 


021 
-077 
235 
-026 
-154 
020 
-099 
-062 
005 
100 
-094 
021 
072 
095 
089 
159 
140 
206 
066 
210 
088 
254 
081 
100 
273 
-038 
134 
194 
-045 
-002 
143 
-026 
054 
-053 
-046 
-254 


111 
005 
093 
-103 
-059 
-125 
-056 
-050 
-021 
-013 
-113 
-058 
-047 
111 
062 
-022 
100 
175 
087 
162 
106 
179 
035 
213 
118 
-125 
257 
100 
-019 
-009 
183 
-041 
055 
-058 
035 
-217 


-133 
-060 
-061 
217 
-040 
166 
189 
034 
302 
083 
237 
043 
230 
043 
081 
-032 
192 
126 
014 
041 
145 
277 
020 
083 
059 
-109 
-012 
-065 
087 
080 
155 
162 
412 
219 
353 
110 


-094 
-101 
-153 
-098 
-039 
-039 
-105 
005 
-079 
-107 
-157 
017 
-031 
017 
072 
-044 
025 
062 

-044 
106 
038 
089 

-072 
090 
046 

-039 

-020 
106 
146 
036 
082 

-087 
036 
128 

-064 

-064 


-034 
-036 
-076 
-078 
-045 
172 
110 
331 
-028 
-038 
-096 
-034 
076 
-034 
071 
-080 
-049 
-150 
-064 
-100 
-075 
092 
-071 
-038 
-073 
-045 
195 
-049 
-137 
163 
139 
-031 
-047 
-034 
-085 
-051 


-084 
-026 
-022 
-121 
030 
154 
091 
072 
143 
-096 
-109 
-084 
-066 
-084 
028 
-132 
110 
-002 
-022 
067 
124 
127 
017 
238 
-086 
-094 
005 
-122 
-022 
Oil 
072 
021 
071 
075 
-123 
034 


-058 
113 
170 
-012 
141 
027 
027 
166 
195 
007 

-099 
-058 
010 

-204 
005 
046 
127 
094 
091 
079 
073 
024 

-005 
007 
096 
084 

-212 

-191 
084 
161 
166 
136 
052 
014 

-056 
199 


-112 
083 
031 
016 
104 
-074 
-113 
018 
021 
055 
-081 
-112 
-010 
-112 
311 
-202 
058 
001 
194 
193 
215 
209 
-051 
123 
062 
-074 
-034 
-052 
101 
023 
070 
-010 
080 
267 
039 
091 


-167 
-031 
098 
041 
-038 
023 
032 
064 
045 
160 
037 
-088 
Oil 
149 
173 
027 
045 
015 
-174 
-065 
068 
179 
-128 
020 
171 
085 
200 
-013 
-041 
243 
336 
016 
064 
070 
113 
-029 


NOTE: 


Correlations  are  phi  coeffldents.   Diagonal  elements  are  squared  multiple 
correlation  which  were  used  as  estimates  of  cowpunali^  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 


E.42 
TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE 
TENCY 


C310 

C312 

C361 

C363 

C364 

C366 

C368 

C371 

C372 

C373 

C374 

C375 

C376 

C378 

C401 

C402 

C404 

C405 

C406 

C407 

C40Q 

C409 

C411 

C412 

C413 

C414 

C415 

C418 

C421 

C501 

C502 

C506 

C507 

C508 

C541 

C542 


I 
1 
1 


C004     COOS     C006  C009 


COMPETENCY 
COlO     con     CO!  2 


026 
068 
-082 
-178 
-091 
-091 
-045 
-055 
097 
-027 
-193 
049 
-164 
-049 
006 
-151 
256 
191 
211 
224 
223 
356 
048 
238 
158 
-149 
203 
038 
009 
060 
048 
138 
060 
-049 
-074 
-089 


-074 
048 
130 
-024 
219 
-099 
-053 
-023 
256 
036 
061 
-074 
-019 
-074 
079 
-030 
-009 
-068 
-058 
050 
138 
202 
-001 
-084 
-007 
007 
-089 
-107 
022 
101 
164 
078 
101 
062 
-045 
080 


118 
-004 
-045 
-169 
-131 
-131 
019 
-154 
-082 
-016 
-117 
-098 
-163 
226 
022 
-059 
094 
-021 
-121 
084 
143 
105 
101 
080 
092 
-131 
156 
094 
115 
026 
-154 
-090 
-055 
010 
-248 
-072 


049 
029 
000 
136 
-034 
176 
036 
-143 
-004 
-069 
122 
-040 
000 
-040 
-134 
-020 
-059 
-234 
024 
014 
010 
039 
-070 
-069 
-080 
036 
-081 
137 
195 
-047 
-143 
073 
156 
-040 
-048 
-007 


-027 
-127 
-159 
-058 
-045 
125 
128 
014 
-035 
-022 
-193 
-100 
-040 
117 
-019 
-116 
013 
-048 
-115 
-010 
060 
233 
019 
-022 
203 
-102 
Oil 
-040 
070 
094 
-036 
008 
-015 
045 
022 
-115 


181 
-120 
-094 
-049 
-150 
-074 
-148 
-109 
-094 
-128 
-076 
-112 
-094 
083 
014 
-057 
122 
-124 
024 
006 
131 
160 
-070 
-128 
087 
079 
113 
-164 
-085 
-157 
-109 
-103 
-157 
-112 
-029 
-101 


-075 
060 
000 
-056 
044 
-020 
010 
-191 
249 
-035 
-083 
-156 
-045 
087 
054 
-111 
008 
-114 
046 
057 
148 
020 
085 
-035 
-066 
107 
017 
-051 
095 
-097 
-079 
030 
148 
006 
-101 
107 


C013     C014     C015   C016   C017  COlJ^ 


-090 
-097 
-016 
102 
-121 
-031 
-167 
096 
059 
-001 
030 
-090 
-016 
-090 
125 
032 
288 
150 
107 
301 
183 
190 
006 
306 
192 
059 
-012 
036 
124 
-040 
175 
040 
134 
140 
Oil 
026 


027 
-009 
-040 
058 
-182 
-012 
051 
-014 
120 
-107 
-170 
-117 
040 
027 
019 
-079 
252 
048 
071 
189 
142 
208 
023 
087 
206 
-182 
174 
-066 
068 
070 
-014 
070 
015 
-045 
-022 
-039 


014 
-024 
050 
-170 
084 
-029 
-018 
-085 
024 
007 
-099 
-058 
-150 
014 
-119 
-111 
074 
094 
135 
151 
113 
245 
-047 
201 
096 
-086 
097 
-032 
187 
106 
-035 
-d20 
052 
160 
-018 
096 


-049  021 
180  157 
072  -Oil 
107  023 
-115  -065 
089  065 
-092 
025 
-012 
-079 
045 
124 


-116 
000 
172 
-019 
-126 
105 
167  126 
037  021 
-058  024 
-044     01 1 


118 
101 
109 


031 
110 
076 


025  062 

083  326 

199  021 

058  -072 

-001  -019 

193  012 

089  -196 

056  -106 


118 
064 


031 
159 


-060  047 
-095  -115 


061 
006 
037 


225 
174 
105 


063  -022 
039     1  34 


03?| 
01 J 

106 
001, 
14tl 
-053* 

-153 

-04r| 

184,1 
-007 
-142« 
0321 
-08?" 
-054 
051| 
082* 
046 
0361 
-05£j 
79 

)5"* 

0831 
-O53J 
-024 
0461 
0421 
-131 
075- 
056| 
128-» 
032 
0491 
-030J 


179 

02- 
0801 
095"^ 
-083 


NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  Mhlch  were  used  as  estimates  of  communal ity  In  the  factor  analysis. 
Leading  decimal  points  are  omitted.  ■  jr* 


] 
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TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPE- 
TENCY 


COMPETENCY 


C603 

C604 

C606 

C608 

C610 

C611 

C613 

C614 

C615 

010 

067 

-079 

104 

-019 

-117 

204 

128 

-015 

-026 

-030 

-141 

072 

-052 

-067 

-031 

212 

-062 

-022 

-047 

-013 

-150 

-044 

-005 

051 

-032 

-206 

(762) 

207 

051 

202 

139 

295 

051 

092 

135 

(713) 

116 

189 

019 

300 

261 

-022 

192 

(835) 

302 

395 

240 

300 

125 

113 

(839) 

206 

256 

144 

259 

170 

(772) 

138 

258 

264 

193 

(759) 

101 

157 

090 

(830) 

131 

253 

(795) 

238 

(711) 

C543 

C545 

C547 

C603 

C604 

C606 

C608 

C610 

C611 

C613 

C614 

C615 

C616 

C617 

C618 

C620 

C622 

C625 

C626 

C627 

C628 

C629 

C630 

C905 

C907 

C908 

C910 

C911 

C912 

C914 

C915 

C916 

C917 

C001 

C002 

C003 


(854) 


356 
(711) 


132 
150 


NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  Mere  used  as  estimates  of  communality  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPETENCY 


XAJ  I  o 

C617 

LO  1 0 

C628 

C629 

C630 

C905 

C543 

209 

067 

-038 

021 

-006 

046 

-179 

030 

-068 

030 

-158 

066 

C545 

055 

023 

115 

102 

004 

-020 

086 

099 

-069 

-217 

-127 

28C 

C547 

no 

-047 

on 

-044 

028 

-141 

179 

135 

111 

-061 

083 

023 

C603 

no 

207 

on 

200 

-054 

243 

099 

135 

-008 

-141 

-no 

023 

C604 

-005 

158 

048 

019 

-005 

058 

-087 

094 

-063 

-014 

-037 

001 

C606 

097 

174 

-044 

227 

267 

252 

080 

314 

095 

246 

004 

139 

C608 

152 

189 

132 

168 

100 

181 

015 

230 

144 

066 

005 

180 

C610 

no 

241 

021 

235 

272 

227 

196 

152 

023 

249 

-154 

196 

C611 

016 

368 

120 

-010 

016 

240 

134 

181 

053 

069 

086 

-014 

C613 

078 

153 

090 

062 

131 

136 

007 

121 

084 

212 

137 

014 

C614 

292 

203 

038 

223 

127 

210 

001 

168 

-029 

108 

-014 

082 

C615 

159 

192 

003 

193 

159 

270 

128 

281 

034 

177 

-004 

007 

C616 

(767) 

069 

-033 

272 

058 

097 

046 

159 

131 

117 

-044 

078 

C617 

(762) 

-094 

-147 

218 

174 

-014 

015 

045 

058 

-037 

155 

C618 

(667) 

124 

-033 

-044 

092 

134 

-060 

024 

-057 

077 

C620 

(775) 

no 

227 

091 

381 

062 

038 

-154 

046 

C622 

(834) 

041 

-024 

-071 

183 

259 

-044 

028 

C625 

(723) 

080 

314 

054 

080 

004 

139 

C626 

(658) 

221 

007 

035 

-051 

200 

C627 

(822) 

121 

221 

061 

132 

C628 

(769) 

161 

137 

-168 

C629 

(757) 

-051 

-045 

C630 

(588) 

-066 

C905 

(780) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


'  4C7 
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TABLE  E7 

Correlations  Among  Package  03  Competencies 


COMPETENCY 


TENCY 

0907 

C908 

C910 

C911 

C912 

C914 

0915 

0916 

0917 

C001 

0002 

0003 

-IT? 

-177 

-OQA 

— U70 

-1AR 

—  1 03 

-177 

AOA 

HUO 

—  1  *tU 

-nQi 

1  1  u 

-nn4 

036 

-047 

-inn 

"  1  o  / 

-170 

-171 

-1A1 

- 1  HI 

70R 
cuo 

-mn 

n3R 

— Ut*r 

nQR 

1  7Q 
1/7 

001 

-1  Vi 
1  ou 

niQ 

-uoo 

-0^ 
—uoo 

071 
u/o 

1AA 
1 00 

U*rO 

1 

uoo 

104 

lUH 

208 

bwO 

\Aj\JO 

OQQ 

uoo 

-01ft 
—uoo 

007 

UU/ 

-040 

•UHU 

-039 

-045 

104 

-038 

W  WW 

LOUf 

_nRn 

010 
u  1  u 

-UHO 

071 
uco 

-077 
•u/  / 

1  S'O 

14A 

1  HO 

014 

UOH 

-ORQ 

— U07 

-197 

PAHA 

nQ7 

1  00 

oil 

u  1  1 

-001 

— UU  1 

07A 

UCH 

110 

1  lU 

07R 
u/o 

uou 

0ft7 

UOb 

04A 

UHO 

083 

070 

w  /  w 

l/DUo 

-1 
1  03 

01 R 

-10A 

~  lUO 

-OftR 
"UOO 

-OAft 
-uoo 

-017 

010 

U  lU 

-OIR 
^uoo 

OAl 
uoo 

7A3 

bOO 

082 

018 

w^O 

USD 

-01 R 

-01 1 
— U  1  0 

-017 
-uo/ 

-010 
—uou 

147 

1  H/ 

OQR 

U70 

0R4 

UOH 

-071 

— UbO 

0Q7 

U7b 

oil 

046 

Pfill 

-116 

1  OQ 

-070 

~U£U 

-OlA 
—uoo 

OOQ 

UU7 

-071 
•u/o 

-OQQ 

— U77 

11R 

110 

107 

lU  / 

144 

140 

147 

Pfil  ^ 

1  O  1 

058 

-0  V 

01ft 
u  1 0 

IPd 
1  OH 

-077 

— Ufc/ 

-04A 

"UHO 

-OQA 

^U70 

-10Q 
—  lu  7 

-70ft 

— bUO 

-OOR 
uuo 

-121 

Pfil  A 

m  7 

-1  AO 

075 

-OilA 

OIR 
uoo 

1AQ 
1 07 

7ftR 

bOO 

70A 

bUO 

041 

UHO 

177 
1  /  b 

118 

237 
b^  / 

PA  11^ 

0^ 

uoo 

113 

101 

lUO 

-Oil 
-uoo 

OAft 

UOO 

-014 

— U  1 H 

-OA  7 
— uo  / 

113 
110 

-ORQ 

1/ J7 

-012 

W  1  b 

PA1  A 

OAA 

-101 
-  lU  1 

0A1 
UH 1 

125 

1RR 
1 00 

OOR 
uuo 

-ORl 
—uoo 

177 
lb/ 

-OIR 
— u  00 

018 

118 

PA1  7 

U90 

lift 

1  1  0 

-OQA 

"U70 

-017 
— uo/ 

-077 

-0A4 

— UOH 

14A 

1  HO 

0Q1 

U7  1 

0Q6 

-070 

027 

Wb  / 

1  AO 

1  07 

-000 

-UU7 

1QA 
1 70 

701 
cuo 

OAR 
uoo 

060 

-0R7 
— uo/ 

01ft 
UOO 

106 

731 
b  «j  1 

030 

www 

1  Ad 

1  OH 

1QA 
1 70 

-1  70 
- 1  /u 

odn 

UHU 

0R7 
uo/ 

17ft 

ObO 

017 
uo/ 

-089 

ORQ 

U07 

0R4 

UOH 

08Q 

006 

wwU 

PA99 

~  1  ou 

-OOA 

-101 
-  lu  1 

-0A7 
-UO£ 

OAA 
UOO 

-077 

— Ufc/ 

117 
01  b 

-ORl 
—uoo 

-038 

016 

-13Q 

1  «j7 

064 

UOH 

-m  A 

— U  1  o 

0?A 
UCw 

-1 R^ 

T  1  OO 

-Ofti 
-UOI 

OAQ 

U07 

1R7 
10/ 

07R 
u/o 

070 
u/u 

-004 

— UUH 

04A 

UHO 

124 

1  b*r 

197 

PA9A 

117 

10A 
1  UH 

-10R 
-  lUO 

070 

u/u 

0R1 
UOI 

-OIQ 
— U07 

-07Q 

-U  /7 

-ORR 
— u  ^0 

-ORl 
— uo  0 

01ft 
uoo 

000 
uuu 

152 
1  ^b 

C627 

15Q 

221 

017 

085 

100 

321 

106 

-049 

168 

074 

335 

160 

C628 

078 

058 

-032 

055 

057 

-027 

-046 

043 

290 

192 

184 

076 

C629 

046 

035 

202 

070 

165 

079 

071 

-054 

-053 

-Oil 

000 

046 

C630 

-044 

032 

042 

-016 

"045 

-098 

-030 

-043 

115 

-117 

068 

108 

C905 

028 

-045 

-019 

-069 

-082 

141 

132 

142 

-031 

023 

-003 

-049 

C907 

(811) 

400 

004 

041 

066 

094 

-071 

139 

-093 

066 

018 

009 

C908 

(785) 

-003 

070 

165 

079 

-154 

-055 

162 

-060 

-051 

-113 

C910 

(730) 

021 

-058 

-133 

240 

245 

234 

014 

040 

-088 

C911 

(784) 

502 

056 

-078 

-105 

148 

-016 

-071 

248 

C912 

(818) 

047 

162 

-075 

035 

-130 

047 

244 

0914 

(846) 

063 

-069 

215 

-024 

183 

158 

C915 

(779) 

154 

052 

074 

11A 

160 

0916 

(785) 

060 

-Oil 

041 

057 

0917 

(816) 

210 

237 

154 

0001 

(762) 

244 

075 

0002 

(739) 

182 

C003 

(721) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  In  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.46 
TABLE  E7 


Correlations  Among  Package  03  Competencies 


COMPt- 

COMPETENCY 

TtNCY 

UJU4 

UJOb 

CU06 

CO  09 

C010 

coil 

C012 

C013 

C014 

C015 

C016 

CO!  7 

coiaj 

C543 

-078 

078 

067 

016 

-032 

-103 

-009 

-009 

-079 

-041 

120 

-Oil 

061-1 

C545 

-159 

111 

-083 

049 

-063 

-195 

-114 

123 

027 

-048 

058 

093 

169 

C547 

143 

219 

-047 

-104 

068 

003 

-083 

149 

045 

-029 

089 

131 

C603 

026 

007 

-047 

036 

-045 

003 

044 

-031 

045 

141 

-047 

131 

014- 

C604 

069 

176 

081 

120 

090 

-032 

107 

-055 

012 

044 

-022 

134 

092  1 

C606 

210 

-102 

-001 

-068 

352 

049 

020 

032 

116 

-032 

002 

056 

036-' 

C608 

-025 

022 

030 

-049 

349 

-046 

-006 

-010 

152 

-059 

027 

113 

085 

C610 

205 

-195 

075 

-021 

284 

156 

003 

-002 

163 

017 

086 

043 

160"' 

C611 

067 

038 

028 

-120 

198 

106 

025 

055 

-016 

006 

-081 

000 

01 Q 

C613 

180 

-075 

045 

032 

061 

022 

120 

-028 

-025 

119 

045 

-084 

158 

C614 

043 

096 

034 

-095 

060 

-131 

196 

071 

167 

058 

017 

022 

228* 

C615 

133 

-127 

089 

118 

073 

-116 

013 

-064 

066 

182 

-014 

-080 

045 

C616 

099 

-023 

069 

-143 

214 

-042 

-079 

-142 

136 

116 

-095 

000 

135* 

C617 

069 

-016 

158 

-070 

039 

-032 

-008 

-137 

064 

044 

-084 

015 

-030 

C618 

254 

-035 

-023 

016 

110 

003 

108 

225 

223 

059 

055 

-027 

102  1 

C620 

013 

084 

-036 

-067 

172 

-045 

086 

175 

089 

092 

042 

086 

204-1 

C622 

-004 

-116 

144 

104 

164 

026 

033 

-142 

036 

-135 

025 

000 

075 

C625 

090 

-030 

058 

-020 

079 

-109 

-111 

032 

038 

085 

049 

-034 

-148" 

C626 

078 

-029 

-014 

089 

-010 

146 

017 

085 

059 

086 

070 

042 

002 

C627 

256 

090 

094 

072 

278 

-021 

067 

288 

093 

127 

-072 

031 

046 

C628 

143 

129 

-009 

-012 

170 

-125 

-001 

087 

120 

083 

-085 

042 

-013-^ 

C629 

128 

-120 

204 

089 

234 

-118 

126 

-147 

-087 

-Oil 

-106 

-183 

002 

C630 

-010 

126 

-037 

053 

041 

016 

064 

072 

018 

002 

038 

017 

03r 

C905 

-088 

-012 

-102 

121. 

-002 

t086 

004 

052 

141 

-Oil 

079 

100 

099 
016 

C907 

151 

-023 

-080 

-019 

-036 

-042 

033 

016 

036 

016 

025 

-173 

C908 

178 

-120 

131 

-091 

039 

014 

072 

085 

059 

-Oil 

070 

-070 

-171- 

C910 

-049 

-074 

010 

049 

-100 

-112 

087 

140 

027 

087 

037 

105 

-140 

C911 

216 

034 

062 

030 

116 

173 

102 

166 

132 

091 

-089 

-010 

106- 

C912 

389 

062 

023 

-089 

100 

077 

-033 

055 

182 

194 

-013 

-093 

071, 

C914 

195 

161 

-077 

146 

081 

-062 

099 

279 

171 

060 

120 

036 

009 

C915 

038 

-009 

015 

007 

013 

-092 

-110 

120 

199 

127 

118 

-031 

172- 

C916 

-104 

-038 

-051 

-068 

005 

-059 

066 

-047 

128 

-144 

-071 

076 

-073 

C917 

160 

167 

-079 

045 

250 

-080 

153 

312 

167 

172 

-028 

284 

-086- 

C001 

004 

137 

-059 

-087 

090 

-062 

024 

216 

153 

270 

-005 

130 

178 
105" 

C002 

202 

257 

-016 

112 

188 

-035 

109 

366 

065 

208 

122 

115 

C003 

183 

160 

027 

-104 

229 

115 

-070 

124 

146 

037 

142 

Oil 

038. 

NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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E.47 


TABLE  E7 

Corral ations  Among  Package  03  Competencies 


COMPE- 
TENCY 


C004     COOS  0006 


COMPETENCY 


C009 

COlO 

con 

C012 

CGI  3 

C014 

C015 

C016 

0017 

C018 

-005 

205 

038 

114 

285 

400 

218 

091 

082 

074 

Oil 

-022 

133 

109 

203 

087 

137 

153 

205 

298 

-007 

142 

107 

165 

-055 

115 

250 

-084 

015 

-030 

(752) 

103 

061 

209 

156 

153 

212 

-019 

049 

-080 

(750) 

132 

021 

-015 

243 

193 

-106 

120 

121 

(688) 

092 

-010 

148 

172 

-014 

013 

032 

(730) 

148 

133 

101 

-054 

089 

071 

(779) 

179 

161 

202 

110 

063 

(773) 

222 

023 

239 

083 

(819) 

-047 

170 

014 

(642) 

072 

144 

(774) 

012 

(716) 

C004 
C005 
C006 
C009 
COlO 

con 

C012 
C013 
C014 
C015 
0016 
0017 
0018 


(825)      105  122 
(745)  -016 
(734) 


NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


410 


E.48 
TABLE  E8 


Factor  Pattern  for  Factor  Analysis  of  Package  01  Competencies 
(Harris -Kaiser  Independent  Clusters  Oblique  Solution) 


COMPE- 
TENCY 

1 

2 

3 

FACTOR 
4 

5 

6 

7 

8 

C201 

201 

318 

-175 

-201 

469 

C202 

-119 

690 

C203 

178 

-142 

151 

-249 

573 

C204 

238 

314 

-123 

121 

367 

C205 

700 

148 

-145 

261 

144 

149 

C206 

-486 

-241 

332 

-135 

-257 

129 

C207 

-263 

111 

-295 

421 

C208 

-755 

145 

161 

-161 

C209 

-164 

-167 

588 

120 

-222 

-224 

C210 

173 

159 

255 

C211 

159 

-105 

331 

488 

223 

C212 

-275 

317 

147 

-333 

179 

-106 

C215 

-125 

-252 

-266 

-274 

341 

200 

C217 

270 

-479 

228 

-140 

C218 

217 

-198 

647 

-174 

276 

C221 

-137 

-145 

-378 

182 

230 

C223 

299 

603 

141 

C301 

354 

-291 

-154 

-169 

288 

C302 

507 

-168 

169 

-186 

-337 

C303 

427 

144 

160 

-154 

C304 

137 

-126 

128 

-484 

110 

C305 

396 

-117 

-372 

-144 

C306 

236 

123 

-433 

100 

C307 

-120 

-209 

-153 

-663 

-232 

C308 

152 

-215 

252 

-217 

-138 

C309 

262 

-122 

147 

-104 

-240 

-129 

129 

C313 

-105 

-230 

-463 

-250 

-260 

C314 

307 

221 

-174 

-270 

C701 

-157 

246 

170 

237 

149 

C702 

191 

196 

407 

207 

-284 

134 

298 

C726 

-428 

-383 

113 

228 

-222 

-287 

318 

C728 

106 

291 

358 

108 

-272 

249 

205 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in  absolute 
value  are  shown. 

Leading  decimal  points  are  omitted. 


E.49 


TABLE  E8 


Factor  Pattern  for  Factor  Analysis  of  Package  01  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


VAJI'lr 
TFNPY 

1 
1 

o 

o 

FACTOR 

A 
'r 

c 
9 

D 

7 

0 

C729 

-124 

160 

707 

118 

-144 

C801 

561 

-209 

-147 

-235 

-168 

187 

C802 

-547 

-138 

-274 

-218 

-101 

-379 

C803 

108 

391 

C804 

282 

163 

-112 

-163 

193 

235 

C805 

-319 

114 

-368 

-204 

322 

389 

C806 

-192 

-443 

355 

C807 

540 

122 

-304 

182 

C808 

141 

-121 

210 

-326 

-321 

168 

295 

C810 

149 

137 

-601 

-203 

C811 

185 

-196 

-127 

706 

-303 

C812 

-110 

-725 

195 

C813 

-341 

213 

-107 

124 

428 

C814 

-462 

174 

-171 

361 

C815 

665 

103 

316 

185 

-174 

C816 

-398 

-221 

-230 

295 

-163 

C817 

115 

272 

-275 

-125 

-283 

185 

-3/  1 

nn 

— ou/ 

-1  79 

"lie 

C821 

107 

-144 

-102 

-290 

-173 

149 

C822 

-228 

-165 

178 

532 

-110 

C823 

-214 

114 

-421 

160 

450 

C824 

471 

-319 

202 

521 

234 

C825 

-539 

-115 

237 

320 

C826 

642 

-251 

-235 

150 

221 

-135 

C827 

-164 

-464 

325 

-163 

C828 

-120 

-600 

129 

134 

-116 

C830 

-536 

237 

-105 

116 

-147 

C831 

-545 

-214 

135 

150 

227 

C832 

-554 

218 

102 

C834 

-102 

322 

-113 

320 

-199 

169 

C911 

-637 

C912 

-141 

-572 

Van  ance 

of  Oblique  4.041 

Factors 

4.429 

2.677 

3.482 

2.454 

3.514 

3.526 

3.040 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in  absolute 
value  are  shown. 

Leading  decimal  points  are  omitted. 


E.SO 


TABLE  E9 


Factor  Pattern  for  Factor  Analysis  of  Packaae  02  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


FACTOR 


0 

0 

A 

c 
9 

D 

357 

-119 

102 

123 

-111 

-132 

252 

328 

211 
253 

-136 

-127 

-209 

207 

-160 

457 

229 

-168 

246 

167 

188 

103 

-293 

186 

119 

266 

116 

-134 

318 

112 

-118 

324 

164 

224 

448 

129 

144 

309 

192 

-220 

-194 

211 
-181 

-131 

-262 

276 

137 

-102 

159 

-139 

-240 
266 

-214 

153 

217 

268 

-140 

-180 

282 

197 

188 

416 

273 

341 

-260 

222 

218 

-340 

116 

392 

-203 

260 

283 

280 

-200 

107 

433 

190 

404 

228 
130 

-260 

-229 

430 

-113 

-123 

401 

-174 

132 

-188 

495 

-139 

-270 

154 

307 

344 

-222 

583 

207 
144 

-143 

278 

142 

-182 

-195 

153 

-176 

-186 

131 

-161 

-332 

111 

299 

-389 

-294 

-578 

435 

-269 

-100 
488 
195 

502 
536 
234 
363 
523 
461 
382 
-118 
310 
362 
707 
500 
454 
413 
"198 


136 


257 
-233 
329 

331 
141 
187 


NOTE:  Only  factor  pattern  coefficients  greater  than  .100  in  absolute 
value  are  shown. 

Leading  decimal  points  are  omitted. 
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E.51 


TABLE  E9 

Factor  Pattern  for  Factor  Analysis  of  Package  02  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


fOMPF- 
TFNPY 

1 
1 

O 

C 

FACTOR 

Q 

4 

5 

6 

ICQ 

441 

-101 

220 

Ol.  I 

1  ^ 

4oU 

-208 

-251 

rfil  1 

-1  Al 

no 

113 

e  p  ii 
-654 

C613 

1  ^fi 

001 

1  CO 

-123 

rfil  A 

vU  1 H 

c>in 
o40 

C615 

QOQ 

lilC 

-l4o 

inc 
lOo 

1  "TO 

-172 

cA^f^ 

Oil  o 

243 

115 

ffil  7 

1  uo 

1  7C 
1  /D 

-494 

ffipn 

vUbU 

CI  C 

ffi?? 

^<>fi 

OQQ 

ooo 

ICO 

168 

1  cc 
166 

-213 

rfi  ?fi 

1  ^fi 
1  00 

-138 

-135 

ffi?  7 

lUD 

OOQ 

226 

1  oi 
131 

-165 

in7 
-  IU/ 

261 

on/1 

329 

-127 

ffi^Q 

0  I  H 

111 
1  1  1 

-lob 

one 

296 

ICC 

379 

1  Al 

070 

117 

197 

ioL 

1  ec 

165 

rand 

17ft 
1  /o 

1  i;q 

no 

324 

-146 

Q7^ 
0/0 

-310 

PQOfi 

-  1 71 

one 

296 

PQin 

vzy  1  U 

01  c 

1  OO 

loo 

C9n 

1^1  (\ 

148 

C91? 

loo 

lyy 

C915 

-180 

191 

275 

160 

216 

C917 

-116 

-153 

605 

215 

-108 

COOl 

-250 

-274 

267 

373 

C002 

-122 

528 

C003 

-277 

353 

155 

343 

C004 

531 

294 

-233 

C0G9 

406 

-118 

218 

183 

COlO 

203 

212 

339 

-124 

con 

114 

300 

-166 

-147 

C012 

325 

194 

-183 

-108 

C013 

198 

514 

-198 

111 

C014 

132 

143 

414 

C015 

121  . 

148 

425 

130 

CX)17 

-126 

477 

-133 

C018 

436 

174 

C019 

116 

-177 

392 

-306 

Van  ance 

of  Oblique  4.790 
Factors 

3.436 

3.794 

3.681 

3.124 

2.981 

NOTE:  Only  factor  pattern  coefficients  greater  than  .100  in  absolute 
value  are  shown.   Leading  decimal  points  are  omitted. 
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E.52 


TABLE  ElO 

Factor  Pattern  for  Factor  Analysis  of  Package  03  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


COMPE- 
TENCY 

1 

2 

PArrriD 
rMi#iui\ 

3 

4 

5 

6 

C310 

137 

-276 

-125 

490 

C312 

713 

C361 

132 

372 

261 

206 

C363 

657 

143 

C364 

-308 

183 

242 

252 

-156 

C366 

205 

364 

C368 

261 

-260 

-112 

220 

105 

C371 

730 

C372 

416 

366 

.  165 

C373 

129 

228 

194 

124 

215 

C374 

540 

300 

C375 

100 

263 

-281 

229 

C376 

437 

-119 

-168 

257 

255 

C378 

267 

-471 

-224 

-219 

-207 

C401 

214 

273 

102 

C402 

116 

389 

132 

-300 

211 

C404 

493 

143 

207 

C405 

412 

190 

134 

C406 

381 

236 

176 

C407 

371 

164 

222 

204 

-211 

111 

C408 

139 

134 

350 

-251 

C409 

495 

112 

C411 

215 

421 

106 

-121 

C412 

488 

167 

203 

C413 

449 

« 

-118 

C414 

-165 

C415 

316 

-134 

-141 

C418 

212 

219 

-173 

-217 

-282 

-120 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in 
absolute  value  are  shown. 
Leading  decimal  points  are  omitted. 
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E.53 


TABLE  E10 


Factor  Pattern  for  Factor  Analysis  of  Package  03  Conpetencies 
(Harris -Kaiser  Independent  Clusters  Oblique  Solution) 


COMPE- 
TENCY 

1 

2 

FACTOR 
3 

4 

5 

6 

1  no 

ZZ3 

-330 

160 

-369 

194 

1  /  / 

506 

CRM 

117 

117 

573 

-  1  00 

261 

23? 

pen  7 

LOU/ 

Z41 

346 

-157 

LOUO 

coo 

lie 

-115 

261 

L34I 

640 

108 

236 

331 

157 

C543 

-1 72 

697 

-170 

-121 

-142 

437 

147 

174 

ruAJ 

188 

-298 

• 

145 

625 

tOU4 

-203 

502 

C606 

-232 

-561 

C608 

-168 

-394 

1  uu 

C610 

120 

-518 

129 

C611 

-373 

-172 

138 

202 

416 

C613 

101 

121 

-399 

-163 

-372 

167 

C614 

236 

-422 

183 

155 

C615 

-423 

-170 

215 

C616 

144 

-432  ■ 

C617 

-150 

-194 

134 

438 

C618 

383 

C620 

192 

-37;^ 

161 

C622 

-189 

-527 

-133 

-218 

-103 

C625 

-390 

221 

233 

C626 

110  - 

-161 

-179 

237 

C627 

143 

-414 

229 

142 

NOTE:    Only  factor  pattern  coefficients  greater  than  .100  in 
absolute  value  are  shown. 
Leading  decimal  points  are  omitted. 
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TABLE  E10 


Factor  Pattern  for  Factor  Analysis  of  Package  03  Competencies 
(Harris-Kaiser  Independent  Clusters  Oblique  Solution) 


COMPE- 
TENCY 

1 

2 

FACTOR 
3 

4 

5 

6 

.pcoo 

1  CC 

-249 

-146 

336 

-484 

-101 

-116 

-130 

LDoU 

110 
-Mo 

1  oc 

147 

-125 

-197 

314 

9A1 

1  CO 

-117 

A^  A 
260 

-fc/o 

TOO 

-123 

123 

269 

1  01 

1  on 

PQll 

111 
-111 

RAH 

-114 

-101 

-108 

1 1<; 

1  ID 

A  Oil 

Oil  ^ 

-243- 

291 

110 

-262 

-225 

105 

-227 

roi  A 

on 
-21 1 

412 

roi  7 
L7I  / 

-220 

501 

124 

rnni 

1 0il 

438 

211 

A^ 

212 

C002 

585 

C003 

159 

^206 

132 

242 

-264 

C004 

530 

-183 

-144 

228 

cods 

128 

527 

C006 

157 

-201 

-272 

C009 

150 

-216 

C010 

-138 

-461 

169 

coil 

175 

-183 

-147 

C012 

318 

-256 

C013 

337 

117 

459 

C014 

313 

-101 

-168 

254 

C015 

208 

-109 

353 

-165 

103 

C016 

144 

154 

C017 

426 

172 

C018 

155 

-136 

213 

-169 

Vari  ance 

of  Oblique  3.973 
Factors 

4.567 

4.153 

3.780 

3.270 

3.189 

NOTE:   Only  factor  pattern  coefficients  greater  than  .100  in 
absolute  value  are  shown. 
Leading  decimal  points  are  omitted. 
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TABLE  Ell 

Correlations  /Vnong  Oblique  Competency  Factors:   Package  01 


FACTOR 

1 

2 

3 

FACTOR 
4  5 

6 

7 

8 

1 

-389 

087 

-335 

004 

-359 

433 

313 

2 

-152 

309 

-103 

280 

-389 

-305 

3 

-024 

009 

-173 

133 

160 

4 

-067 

303 

-364 

-222 

5 

.070 

001 

069 

6 

-383 

-343 

7 
8 

345 

NOTE:   Leading  decimal  points  are  omitted. 


TABLE  El  2 

Correlations  Among  Oblique  Competency  Factors:    Package  02 


FACTOR 

1  2 

FACTOR 
3 

4 

5 

6 

1 

-064 

122 

105 

172 

-029 

2 

126 

193 

007 

-013 

3 

173 

159 

-172 

4 

085 

-055 

5 

-118 

6 

NOTE:    Leading  decimal  points  are  omitted. 
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TABLE  El  3 

Correlations  Anong  Oblique  Competency  Factors:    Package  03 


FACTOR 

1 

2 

FACTOR 
3 

4 

5 

6 

1 

024 

-136 

213 

036 

029 

2 

036 

096 

193 

162 

3 

-121 

-043 

-134 

4 

115 

031 

5 

064 

6 

NOTE:    Leading  decimal  points  are  omitted. 


419 


C002 
C003 


NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communallty  In  the  factor  analysis. 
Le&ding  decimal  points  are  omitted. 


■r,4C6 


E.57 


TABLE  El 4 

Competencies  Selected  from  Packages  01,  02, 
and  03  to  Form  Package  04 


COMPETENCY  NUMBER 


CI  02 

C363 

C610 

CB25 

CI  05 

C371 

C611 

CB26 

CI  06 

C374 

C613 

CB2B 

CI  07 

C404 

C614 

CB30 

CI  09 

C408 

C616 

CB31 

CI  15 

C409 

C617 

CB32 

C116 

C412 

C620 

C911 

C202 

C413 

C622 

C912 

C203 

C415 

C625 

C916 

C205 

.C417 

C629 

C917 

C206 

C420 

C729 

COOl 

C208 

C501 

CBOl 

€002 

C209 

C502 

CB02 

C004 

C211 

C541 

CB07 

C005 

C218 

C543 

CBIO 

C009 

C223 

C603 

CBll 

COlO 

C302 

C604 

CB12 

C013 

C304 

C607 

CB15 

C015 

C307 

C608 

CB19 

C017 

C310 

C609 

CB22 

COIB 

C312 

CB24 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPE- 
TENCY 


C102 
C105 
C106 
C107 
C109 
C115 
C116 
C202 
C203 
C205 
C206 
C208 
C209 
C211 
C218 
C223 
C302 
C304 
C307 
C310 
C312 
C363 
C371 
C374 
C404 
C408 
C409 
C412 
C413 
C415 
C417 
G420 
C501 
C5G2 
C541 
C543 


(741) 


231 
(676) 


CI  06 

CI  07 

COMPETENCY 
C109  C115 

C116 

0202 

C203 

C205 

• 

C206 

C208 

243 

057 

269 

318 

198 

165 

-058 

150 

270 

351 

367 

099 

119 

219 

209 

114 

128 

077 

160 

065 

(774) 

165 

130 

286 

293 

165 

134 

131 

220 

134 

(692) 

230 

096 

190 

105 

110 

001 

051 

110 

(743) 

299 

050 

088 

042 

049 

143 

257 

(828) 

323 

165 

110 

001 

226 

215 

(764) 

-026 

067 

-009 

142 

206 

(793) 

323 

177 

168 

197 

(686) 

015 

-027 

-094 

(851) 

219 

015 

(688) 

431 

(798) 

NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  conmunality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 


E.59 


TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


K^C  1  1 

root 

C302 

C304 

C102 

118 

182 

009 

225 

279 

211 

CI  05 

-030 

057 

063 

113 

113 

035 

CI  06 

248 

156 

120 

070 

120 

047 

C107 

151 

144 

032 

115 

-134 

025 

CI  09 

038 

035 

-006 

108 

108 

078 

C115 

151 

069 

-051 

199 

199 

150 

CI  16 

-087 

185 

123 

123 

013 

169 

C202 

031 

247 

070 

120 

-030 

236 

C203 

-020 

086 

046 

-040 

-127 

-018 

C205 

-105 

229 

-005 

185 

375 

149 

C206 

015 

076 

058 

131 

204 

224 

C208 

263 

008 

307 

133 

133 

244 

C209  . 

(847) 

-112 

295 

-004 

146 

080 

C211 

(722). 

048 

297 

-014 

178 

C218 

(894) 

104 

-034 

-030 

C223 

(770) 

-034 

126 

C302 

(772) 

022 

C304 

(826) 

C307 

C307 

C310 

031 2 

C363 

C371 

C374 

102 

322 

— uo/ 

-1Q7 

-  UDD 

-153 

091 

— uoo 

161 

021 

-077 

-01 1 

198 

042 

-088 

-  1 

-in 

- 1  1  1 

159 

195 

OOR 

-10/ 

- 1  00 

125 

042 

-088 

- 1  til 

-in 

- 1  1  1 

169 

-059 

-064 

040 

-OQ^ 

— UilO 

-uuo 

-057 

244 

202 

015 

-Ofi? 
— uo^ 

uoo 

-005 

-078 

159 

029 

-019 

-025 

049 

029 

-058 

-01  1 
— U  1  1 

184 

010 

-105 

066 

-080 

045 

146 

180 

038 

099 

-123 

036 

137 

240 

108 

204 

082 

232 

187 

-030 

-132 

016 

062 

019 

089 

202 

080 

089 

329 

110 

-091 

100 

-017 

089 

117 

-044 

-031 

100 

080 

-078 

-166 

007 

369 

-100 

-045 

217 

040 

228 

(744) 

-040 

-139 

184 

-019 

167 

(787) 

360 

016 

-101 

-032 

(832) 

072 

-010 

160 

(788) 

049 

566 

(744) 

025 

(770) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


TENCY 

C404 

C408 

C409 

C412 

Cdl 

rdi  7 

PC/11 

Loll 

L54o 

CI  02 

-116 

046 

-089 

-040 

-121 

-126 

-058 

-026 

-103 

-141 

-070 

183 

C105 

024 

-004 

-035 

028 

-029 

065 

-061 

014 

103 

-171 

020 

164 

C106 

-186 

-077 

-010 

-033 

-145 

-055 

-055 

-096 

110 

-062 

187 

366 

C107 

-030 

092 

038 

-094 

-178 

005 

-099 

083 

-115 

-129 

-014 

-032 

C109 

-085 

200 

050 

-088 

-072 

-030 

114 

007 

-200 

-109 

-142 

-040 

CI  15 

-120 

024 

-025 

-094 

-039 

-099 

110 

-110 

-115 

-129 

-079 

094 

CI  16 

-087 

050 

-086 

-068 

-129 

-072 

-072 

-079 

040 

-093 

-065 

091 

C202 

-132 

088 

028 

-165 

-062 

071 

-118 

-038 

-002 

-165 

148 

176 

C203 

-020 

042 

084 

-089 

-024 

015 

015 

097 

086 

055 

140 

119 

C205 

-037 

101 

042 

115 

006 

-065 

-065 

-086 

-096 

-058 

042 

063 

C206 

-064 

083 

-169 

080 

-091 

-118 

-027 

038 

-137 

-154 

-022 

122 

C208 

-114 

114 

-244 

-089 

-096 

-094 

015 

-003 

-109 

-034 

-063 

053 

C209 

-056 

-085 

-023 

-107 

-079 

-114 

-020 

-126 

120 

159 

110 

257 

C21 1 

023 

-016 

103 

'  024 

-006 

-070 

-070 

052 

037 

-002 

172 

094 

C218 

-079 

108 

-022 

-120 

-171 

-127 

-127 

099 

085 

188 

162 

083 

C223 

146 

108 

133 

-029 

-055 

-040 

-127 

019 

007 

188 

108 

135 

C302 

-079 

-006 

-022 

062 

060 

046 

133 

-061 

-070 

-166 

-.160 
-006 

-074 

C304 

023 

078 

-100 

072 

007 

-084 

-149 

-116 

-132 

-066 

074 

C307 

-059 

010 

-006 

091 

-030 

-081 

071 

036 

-114 

043 

045 

016 

C31 0 

-094 

026 

100 

062 

-054 

-078 

-078 

-086 

-090 

004 

075 

-018 

C312 

-101 

085 

04^ 

ndR 

U*tO 

— U03 

-UOH 

n  OA 

1  1  9 

-U 1 U 

1  Oil 

033 

C363 

144 

-Oil 

-007 

123 

-046 

-111 

-181 

-007 

040 

163 

398 

356 

C371 

236 

-050 

066 

071 

016 

-034 

-123 

110 

333 

420 

182 

-006 

C374 

010 

-028 

-035 

061 

-099 

-094 

036 

-008 

145 

131 

330 

321 

C404 

(765) 

038 

259 

288 

172 

169 

075 

135 

036 

159 

052 

-026 

C408 

(715) 

050 

-013 

214 

042 

186 

-059 

-073 

-050 

079 

089 

C409 

(703) 

202 

.  254 

084 

214 

116 

073 

119 

046 

044 

C412 

(748) 

069 

026 

255 

007 

-001 

071 

096 

-060 

C413 

(747) 

122 

267 

-120 

128 

-161 

-041 

-027 

C415 

(731) 

125 

-104 

-012 

-034 

-063 

-078 

C417 

(764) 

-104 

-109 

-123 

-130 

-144 

C420 

(779) 

059 

275 

-032 

-050 

C501 

(750) 

413 

132 

226 

C502 

(831) 

127 

.048 

C541 

(789) 

372 

C543 

(692) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis, 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


C603     C604     C607  C608 


COMPETENCY 
C609  C610 


C611      C613     C614     C616     C617  C620 


037 
072 
-096 
-013 
007 
-013 
-079 
136 
097 
134 
-047 
-104 
-039 
-092 
019 
-061 
099 
064 
-034 
271 
578 
-007 
-054 
-008 
-126 
139 
-004 
113 
-053 
-104 
-104 
-115 
059 
028 
092 
Oil 


270 
318 
062 
-037 
083 
314 
142 
273 
339 
019 
-072 
-027 
015 
-055 
-087 
-014 
204 
059 
-133 
226 
402 
-052 
-080 
-009 
-143 
-037 
105 
-016 
-030 
065 
-027 
-047 
026 
-154 
-079 
067 


-051 
-049 
-190 
037 
-117 
-090 
096 
001 
192 
-219 
-203 
-073 
-077 
-134 
-015 
-173 
-173 
-190 
-159 
065 
259 
-183 
002 
-251 
-077 
-030 
071 
-125 
-017 
060 
-140 
-044 
116 
002 
-145 
-099 


128 
148 
025 
-056 
-151 
066 
099 
135 
172 
175 
030 
-146 
-154 
053 
-187 
116 
015 
041 
-269 
-030 
209 
086 
100 
-082 
-099 
-068 
-041 
-044 
085 
-082 
-209 
-032 
160 
-004 
-133 
104 


042 
-050 
010 
-037 
023 
-125 
-090 
010 
065 
019 
158 
-027 
-143 
-121 
-087 
-014 
-014 
-050 
057 
010 
-105 
183 
-154 
-009 
094 
-037 
050 
-016 
-030 
-118 
-027 
207 
-137 
069 
147 
122 


114 
142 
193 

-090 
188 
228 
012 
078 
126 
169 

-147 

-140 

-134 
056 
038 
038 
144 

-031 

-113 
143 
038 

-098 
056 

-212 
095 
144 
071 
014 
160 

-007 
060 
017 

-120 
052 
059 

-019 


016 

097 
-176 
-070 
-151 
-070 
-004 

058 

266 
-096 
-108 
-140 
-169 
-104 
-060 
-060 
-060 
-027 
-064 

172 

236 
-0'61 

082 
-091 

041 
-151 

075 
-047 
-035 

022 
-140 

220 
-018 

082 
-144 
-117 


306 
228 
241 
075 
184 
260 
240 
204 
230 
130 
261 
036 
-102 
-073 
110 
007 
263 
-081 
077 
197 
089 
027 
-132 
-030 
-102 
-028 
043 
061 
-013 
-029 
-094 
-008 
-143 
-238 
Oil 
204 


056 
097 
-007 
-079 
-098 
-014 
-065 
-046 
140 
091 
034 
-198 
-180 
027 
-053 
108 
054 
-167 
-002 
-084 
049 
139 
127 
-069 
052 
035 
127 
167 
004 
072 
072 
217 
192 
237 
168 
128 


080 
033 
-Oil 
041 
066 
-044 
019 
-013 
-034 
202 
069 
-123 
-071 
-066 
-024 
-024 
046 
040 
043 
004 
-109 
-065 
058 
•132 
-071 
125 
013 
-023 
016 
-034 
-123 
110 
096 
058 
182 
209 


042 

265 
-043 
-125 

023 
-037 

026 

168 

156 

019 

081 
-118 
-143 

010 
-087 

058 
-014 

169 
-007 

226 

199 
-052 

144 
-063 

015 
-037 
-005 

176 
-091 

065 
-118 

122 
-056 

069 
-079 

067 


031 
-049 
001 
-026 
014 
037 
-073 
001 
259 
265 
-092 
-206 
-191 
008 
-173 
038 
-015 
-071 
-159 
-092 
-036 
072 
056 
-055 
152 
144 
031 
084 
027 
-007 
-073 
-044 
057 
110 
182 
021 


NOTE:    Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


TENCY 

C622 

C625 

C629 

C729 

C801 

C802 

C807 

C810 

C811 

C812 

C815 

C819 

Pino 

U2b 

051 

ftftft 

009 

-126 

183 

001 

298 

016 

060 

031 

118 

-026 

PI  nc 
LlUb 

13b 

-Oil 

T  T  O 

113 

065 

A  A 

-025 

-192 

020 

-090 

-084 

-169 

-030 

-138 

LlUb 

l4o 

039 

T  "fft 

170 

197 

A^ 
063 

-078 

-051 

-129 

-132 

-099 

-078 

-085 

LIU/ 

TOT 
12/ 

004 

1  ftft 
199 

110 

-032 

-030 

-051 

086 

061 

016 

151 

060 

onn 
JOO 

024 

ftC  T 

051 

-030 

133 

-023 

140 

-044 

038 

-013 

162 

023 

tl  lb 

212 

1  Oft 

-129 

-099 

031 

-030 

153 

008 

-030 

016 

-120 

060 

pn  c 
tl  lb 

UI9 

n  oc 

03b 

o  oo 

233 

-072 

007 

033 

045 

-107 

152 

-068 

033 

079 

19b 

1  1ft 

119 

ft  *fft 
070 

-055 

252 

-078 

072 

Oil 

-023 

100 

-023 

137 

1  ii  ii 
144 

1  CO 

162 

046 

015 

119 

-114 

-035 

-059 

-114 

141 

-020 

232 

ponc 

072 

171 

122 

-144 

447 

-037 

349 

082 

238 

031 

376 

047 

1  AA 

144 

-117 

131 

-118 

232 

173 

252 

028 

173 

-016 

015 

002 

noil 

-034 

T  ftp 

-185 

046 

015 

119 

263 

248 

104 

075 

026 

169 

001 

-0/1 

-104 

ftfft 
-079 

546 

-083 

-056 

031 

041 

-137 

G90 

025 

-052 

POl  1 

L^l  1 

-002 

OftC 

205 

-014 

086 

282 

091 

279 

128 

158 

271 

158 

145 

POT  O 

C2I8 

-095 

ft  oc 

025 

-103 

394 

083 

071 

042 

198 

071 

062 

445 

161 

117 

1  ftft 
190 

f\  Of* 

035 

046 

030 

-004 

099 

069 

295 

244 

071 

038 

C3UZ 

-095 

1  ftft 
190 

/>  Of* 

035 

-127 

187 

071 

155 

004 

-004 

062 

146 

-023 

C304 

-173 

-089 

-133 

-018 

389 

192 

445 

070 

249 

210 

136 

206 

C307 

-143 

ft  Oft 

-020 

ft  Ol 

-031 

071 

198 

333 

197 

218 

203 

250 

137 

333 

POT  A 

C310 

109 

ftftO 

092 

-003 

051 

-096 

-094 

-063 

172 

-094 

-074 

128 

-035 

C312 

-010 

143 

-113 

039 

106 

-101 

-079 

-034 

-101 

-079 

004 

-050 

C363 

-122 

000 

-133 

099 

-066 

083 

205 

-009 

144 

-024 

023 

003 

C371 

-088 

097 

-095 

323 

102 

-071 

-058 

016 

-148 

164 

005 

177 

C374 

-132 

-028 

-198 

230 

048 

065 

126 

005 

065 

-007 

010 

003 

C404 

-071 

015 

-079 

075 

-083 

106 

031 

-029 

025 

-009 

-056 

081. 

C408 

066 

-068 

-006 

-101 

046 

-023 

093 

009 

286 

062 

162 

023 

C409 

-094 

048 

-074 

084 

005 

-023 

064 

173 

-023 

134 

-023 

117 

C412 

-116 

043 

-120 

026 

147 

189 

-019 

124 

-107 

157 

-107 

017 

C413 

-161 

-010 

-055 

-096 

-114 

-016 

-082 

019 

-204 

-007 

-204 

-069 

C415 

-034 

092 

-127 

-094 

-013 

-114 

-106 

-059 

-020  ' 

-089 

-114 

001 

C417 

-034 

-116 

-127 

-094 

119 

169 

036 

266 

-020 

141 

-114 

155 

C420 

110 

179 

019 

-004 

-050 

-039 

-131 

145 

-039 

219 

048 

-027 

C501 

-142 

156 

007 

280 

050 

-132 

-079 

-018 

-132 

101 

-132 

-040 

C502 

-088 

097 

046 

323 

-059 

005 

-058 

082 

005 

258 

005 

052 

C541 

-093 

082 

001 

275 

-156 

-122 

022 

-093 

052 

-045 

-006 

-130 

C543 

-006 

047 

030 

185 

-032 

-254 

032 

-117 

-026 

-060 

-083 

-201 

NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal i ty  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El 5 

Correlations  Among  Package  04  Competencies 


COMPE- 
TENCY 


C822     C824     C825  C826 


COMPETENCY 
C828     C830  C831 


C832     C911     C912     C916  C917 


C102 

CI  05 

CI  06 

C107 

CI  09 

C115 

CI  16 

C202 

C203 

C205 

C206 

C208 

C209 

C211 

C218 

C223 

C302 

C304 

C307 

C310 

C312 

C363 

C371 

C374 

C404 

C408 

C409 

M12 

C413 

C415 

C417 

C420 

C501 

C5a2 

C541 

C543 


168 
-072 
-022 
108 
024 
-020 
101 
055 
-068 
179 
250 
200 
-128 
305 
152 
312 
-060 
221 
221 
-088 
-031 
084 
009 
076 
045 
200 
019 
020 
-095 
-135 
-001 
085 
-055 
-045 
-093 
-128 


197 
108 
074 
-093 
159 
125 
073 
161 
-081 
368 
247 
222 
-059 
133 
270 
270 
149 
097 
-156 
139 
029 
-107 
-019 
-057 
-059 
209 
-188 
-069 
-130 
-005 
-081 
-034 
-114 
-143 
-095 
-121 


-080 
-057 
-053 
-082 
111 
-082 
-059 
244 
051 
069 
118 
180 
-094 
154 
100 
100 
-003 
-023 
-040 
-064 
-069 
-067 
004 
045 
-094 
111 
-209 
-074 
-140 
180 
-079 
033 
-090 
004 
075 
-096 


181 
-082 
-159 
-098 
047 
-098 
-028 
099 
-012 
353 
173 
213 
-065 
281 
316 
079 
139 
215 
198 
043 
106 
026 
034 
061 
-065 
145 
-036 
005 
-189 
-086 
-012 
097 
-120 
-027 
-061 
-134 


031 
-168 
-161 
016 
-086 
-064 
002 
078 
-053 
034 
038 
030 
-092 
202 
080 
080 
-052 
-Oil 
119 
083 
-029 
-208 
-042 
-223 
-164 
-032 
-088 
-041 
-134 
030 
030 
003 
-083 
-042 
-080 
-253 


191 
-069 
-062 
041 
417 
041 
245 
040 
-034 
137 
144 
412 
005 
189 
117 
■  188 
046 
147 
228 
004 
-010 
-122 
-088 
-132 
-148 
183 
-094 
-023 
-102 
-034 
055 
-136 
-142 
-015 
-093 
-113 


205 
-055 
050 
-057 
203 
024 
222 
245 
038 
108 
190 
377 
-086 
400 
226 
226 
092 
157 
193 
-009 
-023 
-090 
-033 
-010 
-159 
092 
-056 
-124 
-067 
038 
038 
167 
-077 
-033 
-014 
-089 


156 
-102 
-043 
-125 
143 
-037 
026 
115 
065 
019 
158 
339 
173 
141 
276 
058 
058 
224 
184 
010 
-105 
007 
069 
099 
-064 
203 
-169 
-112 
-030 
-027 
-027 
038 
-137 
-005 
-022 
-098 


-031 
-139 
-089 
044 
-077 
-016 
-106 
-162 
-181 
085 
010 
-055 
031 
-250 
-030 
020 
220 
010 
-057 
021 
-077 
-026 
-062 
-094 
140 
088 
254 
100 
273 
134 
134 
020 
-002 
143 
-046 
-165 


-088 
-169 
-201 
161 
-036 
-045 
-041 
-118 
-030 
049 
-097 
-101 
-085 
-222 
-120 
-063 
051 
-136 
-139 
111 
005 
-103 
-050 
-113 
100 
106 
179 
213 
118 
257 
042 
073 
-009 
183 
035 
-127 


-084 
028 
030 
-043 
-075 
-043 
-031 
166 
195 
-062 
-051 
-041 
-049 
105 
132 
-054 
-055 
049 
096 
-034 
-036 
-078 
331 
-096 
-049 
-075 
092 
-038 
-072 
195 
-041 
173 
163 
139 
-085 
-091 


056 
-096 
-163 
-014 
-009 
-144 
192 
-163 
-130 
140 
034 
005 
-122 
027 
054 
162 
054 
156 
-002 
-084 
-026 
-121 
072 
-109 
110 
124 
127 
238 
-086 
005 
005 
154 
Oil 
072 
-123 
-116 


cor!^l.^Hn!lc  !m  f!"'  coefficients.    Diagonal  elements  are  squared  multiple 
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TABLE  El  5 


Correlations  Among  Package  04  Competencies 


COMPE- 
TENCY 

C001 

C002 

C004 

€005 

COMPETENCY 
C009  C010 

001  3 

Till  5 

rni  7 

miR 

CI  02 

-074 

Al 

013 

019 

174 

1 

1  3U 

n7Q 

079 

080 

121 

150 

C105 

-070 

Ail  C 

-046 

-016 

-103 

1  Uo 

-065 

142 

020 

012 

CI  06 

-087 

A^l 

-071 

-148 

-099 

nin 

-002 

091 

-051 

-020 

CI  07 

-117 

007 

-010 

-094 

1 9c; 

1  £3 

1  i;q 

-115 

179 

017 

-178 

CI  09 

-211 

-122 

057 

-013 

1  nn 

-009 

113 

047 

-072 

ens 

-176 

-055 

050 

-094 

a  1 

noQ 

-022 

-058 

-051 

-039 

C1 16 

-020 

A^O 

072 

058 

-068 

OO  1 

1 

040 

214 

045 

-129 

C202 

-016 

004 

-075 

-165 

OTA 

HRl 

-115 

-016 

031 

-020 

C203 

-027 

094 

-050 

026 

-uoi 

ni  1 

-012 

-027 

-035 

-024 

C205 

-01  o 

OCT 

261 

Ail  f\ 

048 

-053 

-01 Q 

117 

-018 

142 

-100 

C206 

-111 

038 

1  AP 

-195 

-016 

I9n 

-UO*f 

107 

-008 

015 

031 

C208 

1  CA 

A  OA 

030 

1 1  ii 
-114 

-089 

1  •«? 

ni  1 

086 

-150 

035 

-024 

C209 

-191 

ICO 

-163 

-235 

-009 

— UOi7 

-048 

-245 

-092 

-141 

C211 

046 

1  Ail 

194 

AT  C 

-016 

024 

PAl 

107 

135 

-076 

-162 

C218 

AC  A 

-060 

AAA 

000 

1  o  o 

-123 

-120 

007 

-158 

042 

-113 

C2Z3 

lo4 

1  AO 

102 

A^A 

078 

062 

1 

uo/ 

239 

086 

267 

-055 

C302 

ICO 

-158 

ACl 

-051 

1  OA 

128 

-029 

-UO!f 

085 

-Oil 

042 

002 

C3u4 

A07 

-09/ 

AT  ^ 

-017 

AAl 

-091 

004 

HAl 

uo  1 

043 

050 

106 

-037 

C307 

-oUI 

AC  O 

068 

AAO 

-092 

-069 

159 

-068 

021 

041 

-049 

-181 

C310 

-Ub8 

no 
-112 

026 

-074 

049 

-027 

-090 

014 

021 

032 

C312 

no 
113 

AOO 

083 

AC  A 

068 

048 

029 

-127 

-097 

-024 

157 

Oil 

C363 

Al  O 

-012 

AT  C 

016 

1  ^A 

-178 

-024 

136 

-058 

102 

-170 

023 

001 

C371 

166 

018 

-055 

-023 

-143 

014 

096 

-085 

000 

-043 

001 

TOO 

uol 

122 

-193 

030 

-099 

-126 

-142 

C404 

127 

058 

256 

-009 

-059 

013 

288 

074 

031 

046 

C408 

073 

215 

223 

138 

010 

060 

183 

113 

326 

023 

C40S 

024 

20» 

356 

202 

039 

233 

190 

245 

021 

080 

C412 

007 

123 

238 

-084 

-069 

-022 

306 

201 

-019 

-083 

C413 

096 

062 

158 

-007 

-080 

203 

192 

096 

012 

083 

C415 

-212 

-034 

203 

-089 

-081 

Oil 

-012 

097 

-106 

-024 

C417 

-150 

-034 

076 

-089 

-005 

-112 

183 

097 

-036 

-024 

C420 

084 

104 

-091 

113 

-034 

182 

-121 

027 

000 

081 

C501 

161 

023 

060 

101 

-047 

094 

-040 

106 

047 

-131 

C502 

166 

0/0 

048 

164 

-143 

-036 

175 

-035 

-116 

076 

C541 

-056 

039 

-074 

-045 

-048 

022 

Oil 

-018 

-022 

049 

C543 

-004 

036 

-078 

078 

016 

-032 

-009 

-041 

-Oil 

061 

NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal Ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


(809) 


207 
(858) 


139 
130 
(773) 


202 
189 
206 
(811) 


C609 

C610 

C611 

C613 

C614 

C616 

C617 

C620 

038 

139 

295 

051 

092 

110 

207 

200 

-149 

019 

300 

261 

-022 

-005 

158 

019 

-092 

074 

335 

101 

018 

-106 

241 

-047 

030 

206 

256 

144 

259 

152 

189 

168 

(683) 

130 

232 

207 

316 

218 

081 

186 

(854) 

138 

258 

264 

110 

241 

235 

(759) 

101 

157 

016 

368 

-010 

(847) 

131 
(765) 

078 
292 

153 
203 

062 
223 

(755) 

069 

272 

(710) 

-148 
(797) 

NOTE:   Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


C622  C625 


C629 

C729 

C801 

1 A1 

-UUo 

1  70 

-014 

-118 

122 

091 

-073 

-099 

066 

-082 

104 

131 

-118 

-154 

249 

060 

061 

069 

-140 

-068 

212 

-158 

048 

108 

005 

128 

117 

055 

155 

058 

-027 

122 

038 

-073 

021 

258 

-034 

-113 

080 

023 

130 

(752) 

-040 

-179 

(812) 

-078 

(832) 

C807 

C810 

C811 

C812 

C815 

C8I9 

UUU 

nA7 

007 

048 

WW  W 

044 

W  1  » 

Ul  0 

-TOP 

— l/U*t 

-112 

lib 

015 

W  1 

131 

-IQl 

-125 

1 

-020 

Wb  W 

040 

w~w 

-U  lU 

-91  7 

—  i9 

-Odd 

-154 

-156 

-045 

01  R 
1/19 

080 

-143 

1     ¥  W 

002 

TOO 

-109 

-077 

Old 

038 

w  wU 

-053 

w  ^  w 

-  IXJO 

-099 

0^8 

-029 

Wb7 

059 

w^^ 

1  oc 
-1^0 

1  Q7 
-  1  Of 

in9 

-075 

010 

003 

Uww 

-066 

lU/ 

-UUo 

1fi7 

-122 

-O^'i 

— 

UDo 

U  1  o 

nA9 

-023 

Wb  w 

005 

WW  w 

-136 

-045 

028 

-064 

080 

-064 

066 

043 

-109 

-077 

014 

-134 

040 

-116 

-116 

005 

-116 

-072 

-074 

-124 

034 

016 

189 

*  016 

042 

-127 

005 

-079 

-120 

-154 

-145 

-036 

022 

-114 

026 

-020 

-076 

459 

275 

258 

286 

314 

402 

153 

251 

188 

189 

106 

480 

(802) 

264 

397 

056 

336 

212 

(806) 

181 

465 

251 

345 

(809) 

090 

431 

214 

(830) 

090 

422 

(851) 

280 

(774) 

-054 

-005 
002 
100 
069 
272 
016 
131 
127 
058 
218 
110 

(712) 


243 
058 
058 
181 
116 
227 
240 
136 
210 
097 
174 
227 
041 
(674) 


-039 
-064 
-020 
-264 

094 
-134 
-029 
-102 
-064 
-148 
-143 
-077 
-148 
-104 
-004 
-114 

258 
(814) 


NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


i 


COMPE- 
TENCY 


C603 
C604 
C607 
C608 
C609 
C610 
C611 
C613 
C614 
C616 
C617 
C620 
C622 
C625 
C629 
C729 
C801 
C802 
C807 
CUlO 

can 

C812 
C815 
C819 
C822 
C824 
C825 
C826 
C828 
C830 
C831 
C832 
C911 
C912 
C916 
C917 


COMPETENCY 


C822  C824 


-100 
-141 
-116 
-040 
027 
-116 
-101 
-082 
113 
-100 
-141 
-035 
118 
-057 
-007 
-135 
235 
333 
271 
247 
391 
301 
218 
191 
(767) 


036 
-070 
-113 
-005 
-133 
117 
-177 
-012 
-142 
-019 
-070 
071 
166 
028 
-031 
-157 
198 
072 
197 
049 
203 
-069 
333 
-041 
360 
(764) 


-086 
010 
-170 
-180- 
010 
-092 
-116 
-032 
-163 
-101 
-098 
-014 
*  004 
Oil 
-105 
-078 
137 
351 
021 
-020 
128 
062 
128 
146 
228 
317 
(860) 


C826 

C828 

C830 

C831 

C832 

C911 

C912 

C916 

C917 

-040 

003 

-054 

-146 

-131 

-038 

007 

-045 

w  »  w 

030 

-015 

-101 

-005 

-094 

005 

-043 

023 

146 

1   •  w 

034 

-080 

-096 

-106 

-082 

-148 

001 

057 

054 

100 

-075 

-143 

-159 

-213 

-236 

-085 

-068 

-036 

063 

-015 

038 

-154 

-023 

005 

-043 

083 

-051 

-079 

-080 

-096 

-106 

-030 

-036 

-037 

-030 

054 

w  w  » 

-023 

V  b  w 

-017 

045 

-116 

-007 

-108 

-036 

009 

115 

107 

-116 

-076 

-027 

-010 

-063 

018 

184 

-096 

-109 

031 

073 

-148 

-067 

-192 

-046 

035 

206 

043 

-027 

-042 

058 

-102 

-005 

041 

125 

-053 

127 

-015 

038 

-080 

-023 

-072 

-096 

-037 

146 

091 

-034 

106 

056 

-133 

019 

040 

057 

-089 

-149 

026 

-088 

-033 

-154 

-062 

067 

-053 

-038 

015 

154 

-016 

172 

-059 

-081 

069 

070 

-004 

-158 

-052 

-095 

-110 

-160 

070 

165 

-055 

-053 

www 

-086 

-136 

-034 

-132 

065 

-055 

-102 

-041 

005 

451 

147 

263 

320 

177 

-051 

-127 

194 

169 

321 

122 

236 

280 

252 

085 

-085 

-049 

110 

472 

014 

231 

193 

252 

-010 

-140 

-077 

109 

371 

291 

016 

183 

028 

105 

-044 

115 

107 

256 

194 

236 

280 

173 

-023 

038 

-049 

227 

240 

394 

164 

232 

272 

-033 

-088 

209 

096 

514 

122 

236 

207 

173 

-023 

-024 

-050 

227 

265 

364 

177 

265 

195 

003 

-078 

266 

155 

478 

217 

227 

289 

306 

-061 

-108 

057 

154 

405 

119 

228 

251 

247 

-057 

-039 

-068 

139 

395 

084 

319 

491 

442 

-202 

-058 

-034 

-004 

(886) 

221 

338 

470 

423 

-202 

-198 

092 

169. 

(705) 

228 

325 

177 

-066 

-032 

120 

124 

(806) 

519 

442 

-165 

-226 

-053 

292 

(860) 

403 

-146 

-186 

126 

143 

(789) 

-201 
(726) 

-217 
502 
(783) 

-051 
-106 
-075 
(706) 

034 
148 
035 
060 
(728) 

NOTE: 


Correlations  are  phi  coefficients.  Diagonal  elements  are  squared  multiple 
correlations  which  were  used  as  estimates  of  communal ity  in  the  factor  ana 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPETENCY 


rnni 
UlU  1 

UJ04 

C005 

C009 

COlO 

C013 

C015 

C017 

C018 

C603 

084 

104 

026 

007 

036 

-045 

-031 

141 

131 

014 

C604 

-059 

038 

•069 

176 

120 

090 

-056 

044 

134 

092 

C607 

092 

050 

244 

154 

-159 

098 

-120 

-021 

043 

-017 

C608 

264 

082 

-025 

022 

-049 

349 

-010 

-059 

113 

085 

C609 

096 

038 

-037 

080 

-133 

142 

107 

096 

074 

335 

C610 

092 

Oil 

205 

-195 

-021 

284 

-002 

017 

043 

160 

C611 

144 

140 

067 

038 

-121 

198 

055 

006 

000 

019 

C613 

-208 

-005 

181 

-075 

032 

061  o 

-028 

119 

-084 

158 

C614 

172 

118 

043 

096 

-095 

060 

071 

058 

022 

228 

C616 

-035 

018 

099 

-023 

-143 

214 

-142 

116 

000 

135 

C617 

096 

-070 

069 

-016 

-070 

039 

-137 

044 

015 

-030 

C620 

054 

089 

013 

084 

-067 

172 

175 

092 

086 

204 

C622 

016 

-139 

-004 

-116 

104 

164 

-142 

-135 

000 

076 

C625 

046 

124 

090 

-030 

-020 

079 

033 

085 

-034 

-148 

C629 

-Oil 

000 

128 

-120 

089 

234 

-147 

-Oil 

-183 

003 

C729 

-027 

-099 

-114 

026 

-081 

-051 

-109 

-027 

-106 

-097 

C801 

-004 

152 

-001 

009 

107 

-106 

050 

108 

075 

-158 

C802 

-085 

058 

038 

-009 

203 

-040 

288 

074 

031 

-016 

C807 

-110 

073 

-082 

130 

197 

000 

047 

010 

035 

-035 

081 0 

052 

045 

-122 

-047 

105 

-167 

055 

052 

-105 

-089 

C811 

-032 

169 

038 

090 

268 

-093 

204 

021 

214 

-079 

C812 

137 

190 

039 

036 

Oil 

-022 

203 

137 

-019 

-007 

C815 

-085 

058 

092 

-008 

072 

-093 

036 

021 

153 

-204 

C819 

-065 

085 

012 

017 

173 

-005 

166 

109 

112 

-120 

C822 

037 

261 

028 

160 

267 

-034 

183 

075 

184 

038 

C824 

-002 

023 

-005 

Oil 

160 

-153 

088 

041 

197 

-130 

C82S 

-131 

040 

026 

062 

049 

-172 

140 

-058 

021 

032 

C826 

-024 

227 

022 

162 

-009 

-171 

013 

018 

230 

-041 

C828 

032 

063 

-154 

-041 

062 

-055 

-010 

032 

014 

-078 

C830 

-085 

227 

048 

071 

166 

-036 

254 

066 

173 

-161 

C831 

-037 

231 

009 

054 

310 

-033 

225 

Oil 

138 

-123 

C832 

-008 

145 

-037 

176 

-007 

-167 

189 

044 

015 

031 

C911 

-016 

-071 

216 

034 

030 

116 

166 

091 

-010 

106 

C912 

-130 

047 

389 

062 

-089 

100 

055 

194 

-093 

071 

C916 

-012 

041 

-104 

-038 

-068 

005 

-047 

-144 

077 

-073 

C917 

210 

237 

160 

167 

045 

250 

312 

172 

284 

-086 

NOTE:    Correlations  are  phi  coefficients.   Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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TABLE  El  5 

Correlations  Among  Package  04  Competencies 


COMPE- 
TENCY 

COOl 
C002 
C004 
COO  5 
C009 
COlO 
C013 
C015 
C017 
C018 


COOl     CQQ2  CQ04 


(750)  244 
(692) 


004 
203 
(815) 


COKPCTENCY 


C005 

C009 

COlO 

C013 

C015 

C017 

C018 

137 

-087 

090 

216 

270 

130 

178 

257 

112 

188 

366 

208 

115 

105 

105 

-005 

205 

286 

218 

082 

074 

(673) 

Oil 

-022 

203 

137 

205 

298 

(723) 

103 

156 

212 

049 

-080 

(773) 

-015 

194 

120 

121 

(851) 

161 
(842) 

111 
170 
(706) 

063 
014 
012 
(707) 

NOTE:   Correlations  are  phi  coefficients.    Diagonal  elements  are  squared  multiple 

correlations  which  were  used  as  estimates  of  communality  in  the  factor  analysis. 
Leading  decimal  points  are  omitted. 
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G.3 


Directions 

In  responding  to  this  instrument,  you  are  asked  to  do  two 

things:    first,  react  to  a  checklist  of  items  by  indicating  the 

level  of  competence  you  feel  you  possess  in  each  specified  area;  and 
second,  indicate  whether  or  not  you  need  additional  training  in  each 
specified  area. 

In  using  the  checklist,  please  keep  in  mind  the  following 
interpretations  for  each  of  the  three  competency  (ability  or  knowledge) 
categories  you  might  choose: 

Hiph  Degree  of  Knowledge  or  Ability:   This  category  refers  to  a 
level  of  competence  that  enables  you  to  proceed  independently  or  take 
primary  responsibility  for  an  activity  in  the  area. 

Moderate  Knowledge  or  Ability:  This  category  refers  to  a  level 
of  competence  less  than  that  in  the  category  above,  but  sufficient  to 
enable  you  to  communicate  intelligently  about  the  area  or  be  a  team 
member  on  an  activity  in  the  area. 

Almost  No  Knowledge  or  Ability;  This  category  refers  to  a  level 
of  competence  insufficient  to  be  classified  in  either  of  the  first 
two  categories. 

In  responding  to  each  of  the  ■  items,  please  make  to*o  check- 
markers,    one  in  Olit  of  the  columns  to  the  left  of  the  item,  and  one  in 
a  column  to  the  right  of  the  item. 
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APPENDIX  I 

TEST  OF  EDUCATIONAL  RESEARCH  AND  EVALUATION  COMPETENCIES: 

FORMS  A  AND  B 


1.3 


DIRECTIONS  FOR  TEST  OF  EDUCATIONAL  RESEARCH  AND  EVALUATION  COMPETENCIES: 

FORMS  A  AND  B 


(Directions  for  use  with  answer  sheet) 

Select  the  best  answer  to  each  question  and  place  the  letter  in  the  corre- 
sponding space  on  the  answer  sheet.    Be  sure  to  black  out  completely  the 
letter  corresponding  to  the  correct  answer. 


(Directions  for  use  without  answer  sheet) 

Select  the  best  answer  to  each  question.  Circle  the  letter  of  the  option 
you  choose. 
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TEST  OF  EDUCATIONAL  RESEARCH  AND  EVALUATION  COMPETENCIES:    FORM  A 


Some  educational  researchers  view  differences  in  aptitudes  as  dif- 
ferences In  time  needed  to  learn  a  task.   Which  educational  practice 
follows  most  closely  from  such  a  conceptualization? 

A.    Providing  a  rich  and  varied  school  curriculum. 

*  B.    Providing  instruction  at  different  rates  for  different  students. 

C.  Providing  periods  of  free  activity  or  rest  interspersed  among 
periods  of  learning. 

D.  Allowing  students  to  pursue  their  individual  interests. 


-3      Questions  2  and  3  refer  to  the  following  problem  situation: 

An  educational  experiment  consisted  of  two  equivalent  groups  (x  and  y) 
taught  by  different  methods  (X  and  Y).   On  the  basis  of  a  test  given 
at  the  end  of  the  experiment,  method  X  (which  was  used  with  the  higher 
scoring  group,  x)  was,  in  this  instance,  judged  by  the  investigators 
to  be  superior  for  developing  the  kinds  of  skills  required  by  the  test. 

For  purposes  of  convenience,  let  I  and  II  stand  for  the  following  ver- 
bal statements: 

I.  "Method  X  is  better  than  method  Y." 

II.  "Group  X  scores  higher  than  group  y." 


Which  pattern  of  deductive  reasoning  has  the  investigator  probably 
followed  in  making  his  conclusion? 

A.  Since  I  is  true  and  since  I  implies  II,  II  is  triie. 

B.  Since  I  is  NOT  true  and  since  I  implies  II,  II  is  NOT  true. 
*C.    Since  II  is  true  and  since  I  implies  II,  I  is  true. 

D.    Since  II  is  NOT  true  and  since  I  implies  II,  I  is  NOT  true. 


*  correct  response 
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3.       Is  the  following  (from  #2  above)  a  valid  example  of  deductive 
reasoning? 

Since  I  is  NOT  true  and  since  I  implies  II,  II  is  NOT  true. 
A.   Yes,  it  is  a  valid  example. 
*B.   No,  it  is  not  a  valid  example. 


4-9      Questions  4-9  refer  to  the  following  lists  of  descriptions  and  names. 
Match  each  description  of  area  of  interest  with  the  name  of  the  scien- 
tist who  has  worked  extensively  in  that  area.   Mark  the  letter  identi- 
fying the  scientist's  name  on  your  answer  sheet  after  the  item  number 
corresponding  to  the  area  of  interest.    No  name  is  used  more  than  once 


Area  Of  Interest 


4. 

*B 

Taxonony  of  Educational  Objectives 

A. 

Alfred  Baldwin 

5. 

*D 

Interaction  Analysis  System  of 

B. 

Benjamin  Bloom 

Classroom  Observation 

Donald  Campbell 

C. 

6. 

*F 

Semantic  Differential 

D. 

Ned  Flanders 

7. 

*I 

Evaluation  Models 

Henry  Kaiser 

e: 

8. 

*E 

Factor  Analysis 

Charles  Osgood 

F. 

9. 

*C 

Sources  of  internal  and  external 

Jean  Plaget 

validity 

G. 

H. 

M.  Rokeach 

I. 

Robert  Stake 

J. 

Paul  Torrance 
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10-12   Questions  10-12  refer  to  the  following  problem  situation: 


There  are  several  ways  to  select  a  simple  random  sample,  although 
some  are  more  efficient  than  others.  For  each  of  the  following  three 
procedures,  MARK  A  on  your  answer  sheet  if  it  will  result  in  a 
random  sample  of  50  persons  drawn  from  a  population  of  80;  MARK  B  if 
it  will  not  result  in  a  random  sample. 


*A  10.    Assigning  each  person  10  numbers  as  follows:  person  1:10-19;  person 
2:20-29;  person  3:30-39;  etc.-  -  then  selecting  the  first  50  persons 
having  at  least  one  of  their  numbers  drawn  from  a  random  numbers  table. 

*B  11.    Selecting  the  person  with  the  next  larger  nuirber  if  the  person  whose 
nunber  was  drawn  from  a  random  nunbers  table  has  already  been  selected 
for  the  sample. 

*A  12.    Arranging  the  80  persons  alphabetically  and  then  selecting  the  50  having 
the  positions  in  the  order  that  match  the  first  50  different  nuirtbers 
read  from  a  table  of  random  nunnbers. 


13.      A  naive  student  constructed  a  questionnaire  to  measure  the  relationship 
between  grade  level  and  library  usage.    One  of  the  proposed  questions 
was : 

How  often  do  you  talk  in  class? 

1.  alwatys 

2.  frequently 

3.  quite  a  lot 

4.  about  average 

Which  one  of  the  following  is  NOT  a  serious  flaw  in  the  student's  .ques- 
tions? 

A.  The  four  categories  are  not  mutually  exclusive. 

B.  The  question  appears  unrelated  to  the  research  problem. 
*C.   There  are  only  four  categories. 

D.  .  The<words  used  are  not  adequately  defined. 

E.  The ^categories  are  not  exhaustive. 


4^3 
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14.      A  test  used  to  select  promising  applicants  for  medical  school  Is 
"valid"  If 

*A.   Those  with  higher  scores  perform  better  In  medical  school. 

B.  Medical  school  students  get  higher  scores  than  those  In  other 
schools. 

C.  It  eliminates  most  of  those  who  apply  for  medical  training. 

D.  Medical  students  get  approximately  the  same  score  on  the  test 
when  taken  at  two  different  times  (about  two  weeks  apart). 


15.      Which  one  of  the  following  practices  Is  considered  MOST  desirable  In 
the  scoring  of  pupil  answers  to  a  series  of  essa^y  questions  In  a  sub- 
ject like  English  literature? 

A.  See  who  the  pupil  Is  so  that  you  can  have  a  better  Idea  of  what 
the  pupil  was  actually  trying  to  communicate  In  his  answer. 

B.  Score  the  answers  for  the  best  and  worst  student  first  so  that 
>^the  range  of  possible  scores  can  be  estimated. 

*C.   Score  each  question  for  the  whole  class  before  going  on  to  the 
next  question  so  scoring  standards  can  be  kept  better  In  mind. 

D.   Score  an  answer  to  the  question  "as  a  whole"  so  that  the  score 
does  not  represent  some  meaningless  total  of  several  part  scores. 


16.      The  correlation  between  the  scores  on  the  odd  and  even  numbered  Items 
on  a  test  Is  .46.   Of  the  options  given,  the  most  reasonable  value 
of  the  reliability  of  the  test  (as  estimated  by  the  split-half  tech- 
nique) Is 

A.  .23 

B.  .46 
*  C.  .63 

D.  .92 
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According  to  evaluation  theorists  like  Robert  Stake,  which  activity 

is  LEAST  essential  for  a  sound  plan  of  evaluating  an  educational  program? 

A.   Obtain  information  which  will  be  useful  in  helping  to  make 
decisions. 

*  B.    Keep  to  a  bare  minimum  subjective  estimates  of  program  objectives 
and  attainments. 

C.  Employ  a  variety  of  data  gathering  instruments  and  measuring 
techniques. 

D.  Describe  the  nature  of  the  educational  program  and  the  context  in 
which  it  is  found. 

E.  Measure  several  unintended    as  well  as  anticipated  outcomes. 


19   Questions  18  and  19  refer  to  the  following  problem: 

Grade  8  math  grades  are  being  used  to  screen  students  for  9th  grade 
algebra  in  Eastern  High  School.   They  correlate  .48  with  algebra  9 
grades.   Scores  from  an  intelligence  test  and  an  algebra  aptitude  test 
administered  in  grade  8  are  both  being  considered  as  a  supplement  to 
grade  8  math  grades  in  the  screening  process.  They  show  the  following 
correlations: 

Algebra        Grade  8 
Grades       Math  Grades 

Intelligence  Test  .40  .50 

Algebra  Aptitude  Test  .40  .  .10 


Which  test  would  be  better  to  add  as  a  second  measure  to  screen 
students  for  9th  grade  algebra—assuming  only  one  test  could  be  used? 

A.   The  intelligence  test. 
*  B.   The  algebra  aptitude  test.  . 
C.    Both  tests  would  be  equally  useful . 

«  ■. '  ' 

Which  correlation  coefficient's )  may  be  correctly  termed  a  predictive 
validity  coefficient? 

A.    Only  .40.  B.    Only  .50.  C.   Only  .48. 

..l.a.    Both  .40  and  .48.  E.   Both  .40  and  .50. 
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20.      The  word  "shrinkage"  is  employed  by  empirical  researchers  in  a 
specific  way.   What  shrinks? 

A.  The  credibility  of  a  theory  when  contrary  evidence  is  presented. 

B.  The  nunter  of  cases  when  subjects  drop  out  of  the  study,  are 
absent,  or  in  some  vay  fall  to  provide  needed  data. 

C.  Test  reliability  when  the  split-half  formula  Is  used  instead 
of  the  test-retest  formula. 

*   D.    Multiple  correlations  when  weightings  are  employed  with  other 
sampl es . 


21-25.  Questions  21-25  refer  to  the  following  paragraph: 

"Faced  with  the  necessity  of  collecting  extensive  data  from  1,452 
recent  liberal  arts  graduates  dispersed  about  the  globe,  this  re- 
searcher devised  a  personalized  approach.  .  .(In  which).  .  .  indi- 
vidual letters  were  prepared  with  the  help  of  computers  and  signed 
by  college  professors  close  to  the  students.   Using  this  appeal 
together  with  a  short  follow-up  letter,  and  a  third  in  some  cases, 
a  return  of  ...  percent  of  the  ...questionnaires  was  achieved  within 
two  months." 


-o  2000 

c 


t  1500 


1/15  1/22         1/29       2/05      2/12      2/19      2/26  3/01 


QUESTIONS:  . 

■■'^  -  -  ■ 

21.      About  what  percent  of  the  questionnaires  were  eventually  returned? 

A.    13       B.   31       C.  56      D.    67        *  E.  93 


I.IO 


22.      The  figure  above  represents  a  cunulative  distribution 
*  A.  True 
B.  False 


23.      The  variable  on  the  ordinate  (vertical  axis)  is  measured  at  the  ratio 
scale  level  of  measurement. 

*  A.  True 

B.  False 


24.  About  twice  as  many  returns  were  received  on  or  before  2/12  than  were 
received  on  or  before  1/22. 

A.  True 

*,B.  False 

C.    Not  enough  information  to  tell. 

25.  If  the  graph  above  were  converted  to  a  histogram  showing  the  nunfeer 
of  questionnaires  returned  each  week>  the  graph  would  be 

A.  Negatively  skewed 

*  B.  Positively  skewed 

C.  Bi  modal 

D.  Bell  shaped 

E.  Rectangular 


26.      Which  statistical  technique  usually  employs  an  independent  variable 
which  is  nominal  (categorical)  in  scale,  and  a  dependent  "variable 
which  is  interval  (equal -interval)  in  scale? 

A.^Contingency  table  (cross-break). 

*  B.    Analysis  of  variance. 


C.  Friedman  test V 

D.  -Multiple  regression. 


lERjq 
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27.       Consider  the  following  proposal  written  by  a  student. 

"This  studly  will  attempt  to  discover  how  repetition  In  grammar 
concepts  through  the  grades  affects  pupil  Interest  In  English... 
Interest  maintained  through  the  grades  will  be  examined  via  two 
questionnaires,  one  given  to  each  English  teacher  and  the  other 
given  to  a  random  sample  of  pupils  In  grades  5-12...  My  limita- 
tions stem  from  the  results  of  the  Interest  surveys;  their  relia- 
bility and  validity  are  difficult  to  assess^ A  random  sample 
of  15  pupils  from  each  grade  level  under  Investigation  will  com- 
prise n\y  sample." 

What  source  of  Internal  Invalidity  presents  the  most  serious 
obstacle  to  this  student  In  accomplishing  her  research  goal  of 
attributing  changes  In  Interest  In  grammar  to  the  repetition  of 
concepts  across  the  grades? 


*  A. 

Maturation 

B. 

Testing 

C. 

Statistical  regression 

D. 

Instrumentation 

E. 

Selection 

28.      Suppose  you  need  to  run  an  experiment  In  which  the  experimental  (E) 
treatment  (after-school  art  Instruction)  requires  the  permission  of 
parents.    What  Is  the  best  procedure,  assuming  you  are  primarily 
Interested  in  Internal  validity? 

A.  Have  alternate  students  standing  by  to  replace  any  In  the  E  group 
who  are  unable  to  secure  permission  of  their  parents. 

B.  Assign  students  randomly  to  E  and  control  (C)  groups.    Let  the 
students  in  the  E  group  who  couldn't  get  permission*  "drop"  and 
assess  this  loss  by  throwing  out  the  data  for  students  in  the  C 
group  who  seemed  like  the  "drops"  in  the  relevant  variables. 

*  C.   Secure  permission  from  all  possible!  parents  before  the  experiment, 
and  before  random  assignment  of  students  to  treatments. 

D.   Assume  that  drop-out  is  random  and  disregard  such  dangers  in  the 
analysis. 


29.      Suppose  you  have  two  classes  available,  both  taught  by  the  same 
teacher,  and  wish  to  test  the  efficacy  of  a  particular  visual  aid 
in  instruction.    For  each  unit  through  the  semester  the  aid  is 
assigned  randomly  to  one  class  or  the  other.   At  the  end  of  a  se- 
mester, student  performance  in  the  units  taught  by  visual  aid  is 
significantly  superior  to  that  in  the  units  taught  without.  Under 
which  of  the  following  assumptions  ma^y  you  generalize  the  results 
to  presence  of  the  aid? 

*  A.   No  interaction  between  treatments  and  any  of  the  specific 
variables  such  as  teacher  or  class,  etc. 

B.  The  teacher  Is  an  average  teacher  and  the  classes  are  average 
classes. 

C.  No  interaction  between  maturation  and  history. 

D.  The  classes,  although  Intact,  are  essentially  equivalent. 


30.      Consider  these  two  designs: 

i.   0  0  0  X  0  0  0  ii.    0_0_0_X_0  0  0 

0  0  0    0  0  0 


An  advantage  of  design  ii  over  design  i  is  that  some  attempt  is 
made  to  control 

*  A.  History 

B.  Selection 

C.  Mortality 

D.  Nultlple-X  interference 

E.  interaction  of  testing  and  X 
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31-33  Questions  31-33  refer  to  the  following  paragraph: 

Tests  1  and  2  correlate  .80.  You  have  Test  1  scores  for  many 
thousands  of  people,  and  from  these  people  you  take  1000,  all 
of  whom  scored  at  point  X  on  Test  1  (see  diagram  below).  You 
now  want  to  predict  their  performance  on  Test  2. 


Test  #  1 


Test  #  2 


X  ■/ 

1 

T        1  1 

1        1 — 

\     i  i 

1                  1                  1  / 

!       !  !/ 
I       1  ^ 

1        1  1 

■ki  1 

1     %   1  I 

1       1    /  1 

I         !  ^  ! 
I         \^  \ 

1      ^1  1 

1      \  1  1 

i  >^  i 

14  1  ! 

t  1<       ^  ^ 

A   B  CD 


*B  31. 

*A  32. 
*D  33. 


A)  Point  A 

B)  Point  B 

C)  Point  C  . 

D)  Point  D 

Which  one  of  the  points  above  do  you  predict  as  the  average  score 
of  the  1000  persons  on  Test  2?  ,  , 

Which  of  the  points  above  would  be  predicted  for  a  correlation  of  1.00? 
Which  of  the  points  above  would  be  predicted  for  a  correlation  of  0.00? 
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34.      Forty-seven  hi*  school  students  responded  to  a  17-item  questionnaire 
on  drugs.  For  each  Question  the  students  were  asked  to  check  SA 
(stronglv  agree),  A  (agree), U  (uncertain),  D  (disagree)  or  SD  (strongly 
disagree).  The  researcher  gave  5  points  for  an  SA  answer,  4  points 
for  jn^A  answer.  3  points  for  a  U  answer,  2  points  for  a  D  answer,  and 
1  point  for  an  SD  answer.   Which  of  the  following  is  the  most  reason- 
able value  for  the  variance  of  the  item  responses  to  a  given  question? 


*  A. 

.47 

B. 

4.7 

C. 

9 

D. 

17 

E. 

47 

35-37  Questions  35-37  refer  to  the  following  data: 

variable^^  ^ol lowing  data  were  accurately  computed  for  some  dependent 


Sex 

Mean 

S.D. 

Diff. 

t 

M 

117.4 

13.5 

1.4 

1.75  * 

F 

118.8 

14.7 

*  p  <  .05  one  tail  test 

For  each  of  the  following  statements  mark  on  your  answer  sheet 

A.  If  the  statement  is  definitely  true 

B.  If  the  statement  is  definitely  false 

^'     i^Ji^^J^^^'l        whether  the  statement  is  definitely  true  or 
definitely  false 

*B  35.    The  correlation  between  sex  and  score  on  the  dependent  variable  is 
zero.  ,( 

*A  36.    The  standard  error  of  the  difference  between  the  means  is  .8. 

*^        Ic%nnLS?i;iIIf?i;'''^^^^^  for  the  true  difference  between  the  means 

IS  approxlmately-r  1.4  to  +  1.4. 
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38.      For  a  group  of  examinees,  the  correlation  coefficient  between  number 
of  items  on  this  test  not  answered  correctly  and  the  total  score  on 
this  test  will  be  closest  to 


A. 

0 

B. 

.30 

C. 

-.30 

D. 

1.00 

*  E. 

-1.00 

39.      A  researcher  used  a  standardized  vocabulary  test  with  a  reported 
(published)  split-half  reliability  of  .90.   When  he  calculated  the 
split-half  reliability  of  the  test  for  the  215  Jefferson  High  School 
students  he  used  in  the  experiment,  the  reliability  was  only  .77. 
Which  of  the  following  is  the  most  likely  explanation  for  the  dis- 
crepancy? 

A.     The  test  did  not  measure  vocabulary  ability  as  well  for  these 
students  as  for  the  norm  population. 

*  B.     The  215  students  were  more  homogeneous  in  vocabulary  ability 
than  the  norm  population. 

C;     The  norm  population  is  larger  in  number  than  215. 

D.     An  error  was  made  since  a  discrepancy  of  this  size  for  such 
large  samples  is  unreasonable. 


40.  A  researcher  has"  a  hypothesis  that  motivation  (M)  affects  learning  (L) 
only  by  triggering  the  aptitude  (A)  of  the.  child.  Which  finding  would 
offer  the  best  support. for  this  hypothesis? 

A.  The  correlation  between  M  and  L  is  zero,   (r^^  =  0) 

B.  The  multiple  correlation  between  L  and  the  combination  of  A  and  M 
equals  zero.   (\  /\m  0) 

C.  The  multiple  correlation  between  L  and  the  combination  of  M  and  A 
is  greater  than  the  correlation  between  L  and  M.    (R^  |y|^  >  r^j^) 

*  D.    The  correlation  between  L  and  M  with  A  partial  led  out  is  equal  to 
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41-42   Questions  41  and  42  refer  to  the  following  problem  situation: 

A  questionnaire  consisting  of  150  items  was  administered  to  a 
group  of  subjects.    The  researcher  wished  to  place  all  the  items 
In  the  questionnaire  into  a  few  scales  which  met  the  criterion 
of  statistical  homogeneity. 


41 


42. 


43. 


The  items  within  such  a  scale  would 

A.  Have  about  the  same  means. 

B.  Have  high  standard  deviations. 

*  C.  Have  relatively  high  correlations  with  each  other. 

D.  Be  statistically  significant  at  the  5%  level. 

E.  Have  about  the  same  item  difficulty. 

The  more  useful  technique  In  accomplishing  this  researcher's  goal 
is  probably 

A.  Cross -tabulation  analysis. 

B.  Analysis  of  variance. 

C.  Bayesian  analysis. 

*  D.  Factor  analysis. 

E.  Nonparametric  analysis. 


A  first  grade  class  has  15  black  and  15  white  children.   A  researcher 
intends  to  select,  at  random,  five  different  children  to  participate 
in  an  experiment.    What  is  the  probability  that  this  selection  pro- 
cedure will  produce  a  sample  of  all  black  children? 

*  A.    Less  than  (1/2)^  ' 


B.  Exactly  (l/2r 

C.  More  than  (1/2)^ 

D.  The  value  cannot  be  determined. 
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44.      Matrix  V  is  a  "singular"  matrix.   This  means  it 
A.    Is  a  scalar. 
*  B.    Has  no  inverse. 
C.    Is  an  identity  matrix. 
0.    Not  necessarily  any  of  these. 


I 


I 
1 


45.      Consider  the  following  to  be  a  sequence  1n  a  FORTRAN  program. 
What  will  N  be  equal  to  at  the  end? 

N  =  0 

DO  20  I  =  1 ,  10 
20  N  =  N  +  1 

A.  0 

B.  1 
*  C.  10 

D.  11 

E.  20 


■Mm 


TEST  OF  EDUCATIONAL  RESEARCH  AND  EVALUATION  COMPETENCIES:    FORM  B 


1.       Assume  that  It  Is  true  that  measurements  of  brain  activity,  rapid 

eye  movement,  and  muscle  activity  show  specific  patterns  during  dreams 
which  are  absent  during  non-dream  periods.    In  contrast  with  a  re- 
searcher who  ooesn't,  a  researcher  who  measures  this  activity  and 
movement  my  be  able  to  provide  direct,  objective  evidence  about  the 
likely  validity  of  which  one  of  the  following  hypotheses? 

*  A.   More  Intelligent  children  dream  more  than  less  Intelligent  chil- 
dren. 

B.  Children  who  have  a  tension-filled  school  day  will  have  more 
unpleasant  dreams  than  children  having  happy  sfhool  days. 

C.  Dreams  represent  real  desires. 

D.  There  is  a  negative  relationship  between  pleasantness  of  dreams 
and  accuracy  with  which  the  content  of  dreams  is  recalled. 


2.       The  reason  that  "fate  caused  It  to  happen"  Is  not  used  as  a  scien- 
tific explanation  Is  because 

A.  Researchers  don't,  as  a  group,  believe  In  fate. 

B.  Hystlcal  concepts  are  excluded  from  scientific  discussion. 

*  C.    Fate  can  be  used  to  explain  why  anything  happens,  and  thus  falls 
to  explain  why  this  rather  than  that  happens. 

i 

D.    Fate  refers  to  something  which  Is  not  observable  In  the  empirical 
sense. 
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3-8     Questions  3-8  refer  to  the  following  paragraph: 

A  naive  student  researcher  conducted  a  stu4y  In  which  he  wished  to 
assess  the  degree  of  relationship  between  certain  leadership  dimen- 
sions for  principals  and  their  pupils'  attitudes  toward  school.  He 
used  a  commercially  available  pupil  attitude  Instrument  and  construct- 
ed his  own  measures  of  the  leadership  dimensions.   The  studly,  as 
conducted,  had  some  undesirable  characteristics  that  would  concern 
an  expert  researcher  as  well  as  some  characteristics  that  would  not 
be  of  much  concern.   For  each  of  the  six  characteristics  listed  below, 
Indicate  whether  the  feature  should  be  of  concern  or  not.  (Note: 
Three  Items  are  keyed  A,  three  B.) 


3.       The  measure  of  principal  leadership  style  was  not  pretested. 
*  A.   Undesirable  feature. 
B.   Should  not  be  of  concern. 


4.       Only  10  principals  were  used. 
*  A.   Undesirable  feature. 
B;   Should  not  be  of  concern. 


5. 


I 
I 
I 
I 
I 
I 
I 


6. 


7. 


The  pupil  attitude  Instrument  did  not  have  adequate  norms. 

A.  Undesirable  feature. 

*  B.   Should  not  be  of  concern. 

rThe  names  and  attitude  responses  of  pupils  were  shown  to  the  respec- 
tive principals. 

*  A.   Undesirable  feature.  / 

B.  Should  not  be  of  concern. 

No  responses  from  teachers  were  obtained. 
A.    Undesirable  feature. 


B.   Should  not  be  of  concern. 


8.       tornrorcial  and  home-made  instruments  were  mixed  together  in  a  single 
stucly. 

'        A.   Undesirable  feature. 
;     *  B.  Should  not  be  of  concent 
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9-10     Question:,  9  and  10  refer  to  the  following  paragraph. 

An  investigator  has  developed  a  scale  which  he  hopes  is  a  valid 
measure  of  "severe  anxiety".   He  correlates  this  measure  with 
measures  of  educational  achievement,  sex.  and  degree  of  inconsis- 
tency in  child  rearing  practice.   The  correlations  are  -.34.  .02. 
and  .56  respectively. 


9.       The  investigator  was  probably  trying  to  demonstrate 
*  A.    Construct  validity. 

B.  Face  validity. 

C.  Validity  by  the  multi -trait,  multi -method  procedure. 

D.  Predictive  validity. 

E.  "  Concurrent  (status)  validity. 


10.      For  his  intent,  the  findings  tend  to 

*  A.  Be  supportive  of  the  measure's  validity. 
B.    Fail  to  support  the  measure's  validity. 


11.      Which  one  of  the  following  practices  would  help  the  most  to  insure 
that  a  researcher  obtains  reliable  ratings  of  certain  traits? 

A.   Include  scales  of  traits  that  are  independent  of  each  other. 

*  B.   Provide  definitions  or  examples  of  the  points  on  the  scales. 

C.  Include  at  least  five  (7  or  9  preferable)  points  on  each  of  the 
scales. 

D.  Instruct  raters  to  stick  with  first  impressions  and  to  resist 
changing  ratings  once  made. 
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12.      What  is  the  principal  objection  to  broad,  generalized  statements  of 
instructional  objectives  such  as  "to  develop  good  citizenship?" 

A.  Such  objectives  are  not  as  important  as  the  intermediate,  more 
narrowly  defined  skills. 

B.  In  reality,  little  instructional  effort  is  directed  toward 
teaching  for  such  objectives. 

*  C.   The  evidence  that  such  objectives  have  been  attained  is  not  de- 
fined. * 

D.   They  are  usually  taken  for  granted  and  tell  little  about  the 
distinctive  characteristics  of  a  school  system. 


i 


13.      Who  of  the  following  is  not  a  well-known  writer  in  curriculum  evalu- 
ation? 


*  A.  Ramsey 
D.  Tyler 


B .   Scri  ven 


C.  Stufflebeam 


14. 


I 
I 
I 
I 


15. 


For  purposes  of  curriculum  evaluation,  human  judgments 

A.  Are  not  usefully  quantifiable. 

B.  nay  be  expected  to  conform  with  test  evidence. 

*  C.    Can  be  usefully  summarized  by  available  sampling  and  statistical 

techniques. 

D.    Should  be  gathered  mostly  from  the  classroom  teachers. 

In  order  to  obtain  a  critical  appraisal  of  the  Stanford  Achievement 
Test  Battery,  which  one  of  the  following  resources  is  likely  to  be 
most  helpful? 

*  A.   Mental  Measurements  Yearbooks 

B.  Education  Index 

C.  Educational  and  Psychological  Measurement 

D.  Encyclopedia  of  Educational  Research) 


E.   Tests  In  Print 
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16.      A  student  receives  a  test  score  of  58.    Which  one  of  the  following 
would  give  you  the  most  information  concerning  how  man^  points  this 
student's  score  is  likely  to  differ  from  his  "true"  score? 

*  A.    The  standard  error  of  measurement  of  the  test. 


C.  The  standard  deviation  of  the  test  scores. 

D.  The  validity  of  the  test. 

E.  The  z  score  (or  T  score)  equivalent  of  a  test  score  of  58. 

17.  Most  of  our  test  predictions  of  school  success  are  based  on  correla- 
tions of  about 

A.  .00 

B.  .20 

*  C.  .50 
D.  .  .80 

18-19  Questions  18  and  19  refer  to  the  following  paragraph: 

You  wish  to  predict  an  adolescent's  success  in  engineering.    He  takes 
two  tests  that  are  called  "aptitude"  tests  to  find  out  whether  he  has 
an   aptitude"  for  engineering.   On  Test  #1  he  scores  a  z  score  of +2.00. 
On  Test  #2  he  scores  a  z  score  of  -1.50.   You  find  that  both  tests  are 
correlated  with  engineering  success  but  the  correlations  are  different. 
Test  #1  correlates  with  engineering  success  .15,  while  Test  #2   has  a 
correlation  with  success  of  .75. 

18.  In  making  your  prediction  about  the  adolescent's  future  success  in 
engineering,  you  should 

A.  Consider  both  tests  of  almost  equal  merit. 

*  B.    Practically  ignore  Test  #1. 

C.  Practically  ignore  Test  #2. 

19.  On  the  basis  of  the  evidence  presented  above,  you  would  predict  that 
the  adolescent  would  probably 

*  A.   Do  poorly  in  (engineering. 

B.  Do  about  average  In  engineering. 

C.  Do  well  In  engineering. 


B. 


The  reliability  of  the  test. 


D. 


It  would  be  impossible  to  s 


cause  of  conflicting  test  results. 


1.23 


20-     Suppose  you  want  to  study  the  effect  of  treatment  x  on  a  single  trait 
measured  by  different  tests  0^  and  Og.    Here  are  5  possible  designs. 

(E  and  C  represent  experimental  and  control  groups  respectively.) 
I-     E:XOa  II:     E:  0^  X  Ob  HI.     E:  0^  X  Oa 

C-     Oa  C:  Oa     Og  C:  Og  Ob 


°A     °B  V.       E:  Oa  X  Og 

C:  Og  X  0^  C:  Og  0^ 

Which  design  is  best  if  sifcjects  were  NOT  randomly  assigned  to  the 
experimental  and  control  groups? 

A.  Design  I 

*  B.  Design  II 

C.  Design  III 

D.  Design  IV 

E.  Design  V 


21.     Annual  achievement  tests,  illness  records  daily  assignments,  and  other 
routine  characteristics  and  activities  of  the  school  are  considered 
for  experimental  purposes. 

A.  Reactive. 

*  B.  Nonreactive. 


22.     In  classrdom  experiments  where  achievement  in  a  subject  is  the  criterion, 
which  solution  best  answers  the  problem  of  biased  assignment  and  unique 
intragroup  histories? 

*  A.     Moving  the  randomization  to  classrooms  as  units. 

B.  Setting  up  randomization  procedures  to  assign  students  to  classes 
at  the  beginning  of  fall  semester. 

C.  Applying  analysis  of  covariance  to  individual  student  ability. 

D.  Use  of  the  within-class  variance  as  error  term. 


"480 
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■25.    Questions  23-25  refer  to  the  following  problem  situation: 

Six  students,  three  first  graders  and  three  second  graders,  are 
each  administered  a  test  twice,  once  after  exposure  to  a  control 
treatment  and  oi:ce  after  exposure  to  an  experimental  treatment. 
Let  G  represent  grade;  S,  student;  T,  treatment;  and  X,  test  score. 
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Which  diagram  is  NOT  an  accurate  portrayal  of  t);a  research  design? 

A.  Diagram  A  C.     Diagram  C 

B.  Diagram  B  D-     Diagram  D 

Which  one  of  the  following  does  NOT  accurately  describe  a  character 
istic  of  the  design  as  described  verbally  above? 

A.  Treatment  is  crossed  with  grade. 

B.  Treatment  is  crossed  with  student. 

C.  Student  is  nested  within  grade 

D.  Grade  is  nested  within  treatment. 

These  data  would  probably  be  analyzed  by  using 

A.  An  analysis  of  covariance  model. 

B.  A  fixed  effects  analysis  of  variance  model. 

'  C.     A  repeated  measures  analysis  of  variance  model. 

D.  A  random  effects  analysis  of  variance  model. 

E.  A  non-linear  analysis  of  variance  model . 

What  is  the  median  of  the  following  numbers:  9,2,  1 ,  9,  4? 
A.    1        *  B.    4  C.    5  D.  9 


482 
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27-29    Questions  27-29  refer  to  the  following  problem: 

A  test  has  a  mean  of  100  and  a  standard  deviation  of  16.  Joe  scored 
84  on  this  test. 


27.  What  is  Joe's  score  in  z  score  units? 

A.    +16.00  B.  +1.00  *C.  -1.00  D.  -  16.00 

28.  What  is  Joe's  score  in  T-score  units? 

A.    85  B.  65  C.  50  *D.  40 


29.       Assuming  the  test  score  distribution  is  normal,  what  percentage  of 
the  population  will  have  scores  lower  than  Joe's? 

A.    2%  B.  S%  *  C.  16«  D.  35% 


30-32    Questions  30-32  refer  to  the  following  paragraph: 

Suppose  that  a  30-in1nute  test  of  reading  speed  and  a  30Hn1nut6  test 
of  comprehension  appropriate  for  use  In  grades  7-9  have  been  given 
to  100  children  from  grade  8  only,  and  the  correlation  between  these 
two  tests  has  been  found  to  be  r  ■  .60. 


30.       If  there  had  been  300  children  In  the  sample  Instead  of  100,  but  still 
of  the  same  type,  grade  And  age  we  should  expect  that  the  correlation 
would  be 

A.  Larger  than  .50. 

B.  Smaller  than  .60. 

C.  Precisely  .60. 

*  0.   Near  .60,  perhaps  .  Mttle  larger  or  a  little  smaller. 
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If  the  tests  described  in  the  preceding  question  had  been  given  to 
100  children  from  grades  7,  8  and  9,  we  should  expect  that  the  cor- 
relation would  be 

A.  Larger  than  .60. 

B.  Smaller  than  .60. 

C.  Precisely  .60. 

D.  Near  .60,  perhaps  a  little  larger  or  a  little  smaller. 


If  the  tests  described  above  were  reduced  in  length  by  eliminating 
about  half  of  the  questions  at  random,  we  should  expect  the  correla- 
tion between  the  shortened  forms  to  be 

A.  Larger  than  .60. 

B.  Smaller  than  .60. 

C.  Precisely  .60. 

D.  Near  .60,  perhaps  a  little  larger  or  a  little  smaller. 


Assuming  thato~x  constant,  as  a  correlation  increases  from 

.0  to  1.00,  the  standard  error  of  estimate 

A.  Increases  also. 

B.  Remains  approximately  the  same  size. 
*  C.  Decreases. 

D.  Impossible  to  say. 


A  class  of  30  entering  kindergarten  students  were  asked  individually 
to  do  two  things:  name  the  color  of  a  brown  circle  and  pick  a  triangle 
from  a  number  of  shapes.   Twenty  children  could  name  the  color,  15  could 
pick  out  the  triangle,  and  12  children  could  do  both.   The  relationship 
between  whether  or  not  children  are  able  to  name  "brown"  and  whether 
or  not  they  are  able  to  pick  out  the  triangle  could  most  appropriately 
be  shown  by  computing  which  correlation  coefficient? 

A.  Eta 

B.  Point  biserlal 
*  C.  Phi 

D.  Kendall 's  tau 

E.  Spearman's  rho 


■  m 
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35-36       Questions  35  and  36  refer  to  the  following  set  of  figures: 

Shown  below  are  four  possible  ways  In  which  two  predictors, 
P«l  and  Pg,  could  correlate  with  each  other  and  with  a  criterion 

variable  C. 


(A) 

PI  P2 

C 

PI 

1  .5 

.5 

P2 

1 

.5 

C 

1 

(B) 

PI     P2  C 


1  .0 

.5 

1 

.5 

1 

(C) 

PI  P2 

C 

1  .0 

.0 

1 

.5 

1 

(0) 

PI  P2 

C 

1  .5 

.5 

1 

.0 

1 

35.    Which  martrix,  used  In  multiple  regression,  will  best  predict  the 
criterion? 


A.  A 


*  B.  B 


C.  C 


D.  D 


36.    Which  of  the  matrices  above  best  exhibits  what  Is  called  a  "suppressor 
variable?" 


A.  A 


B.  B 


C.  C 


*  D.  D 


37.    In  a  principal -components  factor  analysis,  the  correlation  of  a  test 
with  a  factor  is  called 

A.  Connunality. 

*  B.  Loading. 

C.  Centroid. 

0.  Rotation. 
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A  researcher  seeks  to  compare  the  effect  of  two  different  counseling 
treatments,      and  Tg.    As  each  counsel ee  is  to  begin  treatment,  an 

unbiased  coin  is  flipped  to  determine  whether  to  use      (a  head  appears) 

or      (a  tail  appears).    Suppose  the  last  three  counselees  were  all 

assigned  to      (because  tails  appeared  in  each  case).    The  probability 

that  the  next  student  will  also  be  assigned  to  the  T,  treatment  is 


n 

Somewhat  less  than  that  of  being  assigned  to  the 

procedure. 

B. 

Exactly  the  same  as  that  of  being  assigned  to  the 

prcceuure. 

C. 

Somewhat  more  than  that  of  being  assigned  to  the 

procedure. 

D. 

One  in  sixteen. 

E. 

Not  determinable. 

If  the  difference  In  means  of  an  experimental  and  a  control  group  is 
statistically  significant  at  the  5«  level,  one  may  conclude  that  the 

A.  Odds  are  at  least  19  to  1  against  such  a  difference  arising  from 
sampling  error  under  a  true  null  hypothesis. 

B.  Mean  difference  was  at  least  20  times  as  large  as  the  standard 
deviation. 

C.  Power  of  the  statistical  test  was  .95. 

D.  Probability  that  the  null  hypothesis  was  true  was  at  least  .95. 

E.  Experimental  treatment  was  much  more  effective  than  the  control 
as  measured  by  the  dependent  variable. 

In  testing  for  the  significance  of  the  difference  between  two  means, 
an  Investigator  failed  to  use  the  formula  which  takes  Into  account  the 
fact  that  both  means  were  obtained  from  the  same  sample  of  85  persons. 
Repetitions  of  this  error  will  cause  the  Investigator  to 

A.  Announce  too  few  significant  results. 

B.  Announce  too  many  significant  results. 

C.  Reject  too  many  true  null  hypotheses. 

D.  Underestimate  the  standard  error  of  the  difference  between  means. 

E.  More  than  one  of  the  above. 
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Questions  41  and  42  refer  to  the  following  problem  situation: 

The  following  table  shows  the  mean  scores  on  a  test  of  "fate  control" 
for  four  large  and  equal  sized  groups  of  black  and  white  pupils  in 
two  grades.   A  high  "fate  control"  score  means  belief  that  most  events 
are  caused  by  fate.    Data  are  hypothetical. 


GRADE 


RACE: 


Black 
White 


4th 


8th 


14.7 

6.1 

10.5 

6.0 

Which  of  the  following  conclusions  can  be  correctly  drawn  from  the 
table? 

A.  More  education  is  responsible  for  people  believing  less  in  the 
importance  of  fate. 

B.  The  effects  of  race  and  grade  are  additive  (in  a  negative  sense). 

C.  Fourth  graders  believe  fate  is  roughly  twice  as  important  as 
eighth  graders  do. 

D.  Grade  and  race  interact  in  a  statistical  sense  in  relation  to 
"fate  control"  measures. 

Which  statistical  technique  is  most  appropriate  for  analyzing  data 
available  from  such  a  study? 

A.  t  test. 

B.  Chi -square  test. 

C.  Analysis  of  variance. 

D.  Factor  analysis. 

E.  Phi  correlation  coefficient. 


I 
I 
I 

I 
I 
I 
I 

i 

II 
H 

li] 
G 
I! 
E 
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43.  Consider  the  following  FORTRAN  expression: 

Z  =  X  **  Y  *  1/Y. 
If  X  »  3  and  Y  «  2  what  does  Z  equal? 

A.  3.0 
*  B.  4c5 

C.  6.0 

D.  6.5 

44.  You  are  given  two  simple  matrices: 

2  ^ 


S  = 


1 


T  = 


A  third  matrix  X  =  S  •  T.  What  Is  the  value  of  X  (1,1)? 
A.    12  B.    18  *  C.    39  D.  75 


45.       In  flow-charting  for  computer  programs,  the  symbol 


usually  means 

A.   Input.         B.   Output.         C.   Operation.         *  D.  Decision 
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PART  II.     THE  DEVELOPMENT  PROCfS^ 


Directions 

This  part  of  the  examination  consists  of  34  items.    The  first  18 
items  relate  to  two  brief  descriptions  of  segments  in  the  product  develop* 
ment  process.    Each  description  is  followed  by  nine  statements,  some  of 
which  are  correct.    You  are  to  read  each  description,  then  on  the  answer 
sheet  which  has  been  provided,  mark  an  A  for  each  statement  which  is  cor- 
rect and  a  B  for  each  statement  which  is  incorrect.    If  insufficient  infor- 
mation has  Feen  presented  for  you  to  judge  the  correctness  of  a  statement, 
then  leave  the  item  blank. 

Generally  speaking,  certain  incidents  in  the  development  of  instruc- 
tional products  are  recounted  in  the  fictitious  descriptions.    Your  task 
is  to  identify  correct  procedures  which  were  employed  or  errors  which  were 
made  by  the  product  developers.   No  attempt  has  been  made  to  be  devious 
in  the  examination.    You  need  not  "read  between  the  lines"  in  order  to  judqt 
the  correctness  of  the  statements.    The  answers  should  be  apparent  to  you 
if  you  are  familiar  with  appropriate  steps  in  the  product  development  pro- 
cess.   Be  sure  to  use  the  answer  sheet  for  your  :^<tspc«:ses.    Now  commence 
with  the  first  description  and  its  accompanying  nine  statements. 

Exercise  One 

It  is  mid-December  and  Frieda,  a  regional  laboratory  employee,  has 
been  given  the  responsibility  of  developing  a  short  self-instruction  program  to 
teach   sixth  grade  pupils  how  to  use  commas.    She  has  received  a  set  of  fr^e 
explicit  instructional  objectives  from  the  individuals  who  originally  form- 
lated  the  project.    Along  with  these  objectives  there  are  samples  of  a  single 
prototype  test  item  for  each  objective.    Frieda  has  been  told  that  the  instruc- 
tional product  is  to  take  no  more  than  four  hours  of  the  average  learner's  time. 

The  first  thing  she  does  is  to  develop  a  40-item  criterion  test  to  be 
used  at  the  close  of  the  program.    She  has  each  of  the  five  objectives  repre- 
sented by  at  least  five  test  items*  although  two  objectives  which  she  feels 
to  be  more  important  are  represented  by  10  items. 

Frieda  carefully  considers  the  enroute  (intermediate)  behaviors  which 
the  learner  must  master  on  his  way  to  the  criterion  behaviors  and  then  sequences 
these  from  least  to  most  difficult »  ending  with  behaviors  like  those  called 
for  in  the  instructional  objectives.    She  then  prepares  practice  sequences 
for  each  of  the  enroute  and  terminal  behaviors  so  that  the  learner  will  be  abl^ 
to  practice  using  ccnmas  in  o.  variety  of  situations.    After  having  four  col- 
leagues react  to  her  first  version  of  the  Instructional  product,  she  makes  a 
number  of  revisions. 
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Frieda  then  arranges  to  field  test  the  program  in  the  public  schools 
and  secures  the  cooperation  of  a  nearby  elementary  school.   She  arranges 
to  use  three  classes  for  approximately  oiYe  week  and  adii?1n1sters  the  program 
in  early  February.   At  Its  conclusion  the  40-item  criterion  test  Is  given 
to  each  of  the  136  children  who  completed  the  program. 

Frieda  Is  pleased  that  all  youngsters  were  able  to  finish  the  instruc- 
tional product  In  three  hours  or  less.    She  Is  somewhat  concerned,  however, 
that  the  average  score  on  her  40-1tem  test  Is  only  21.2  correct.    She  resolves 
to  study  the  post-test  results  as  well  as  the  responses  made  by  pupils  during 
the  program  and  to  make  the  revisions  which  seem  dictated  by  the  data. 

1-9   For  Items  1  through  9  mark  A  If  the  statement  Is  correct i  mark  B  if 
It  Is  Incorrect. 

*A  1.  Frieda  was  probably  correct  In  providing  practice  behaviors  for  the 
learner  which  were  like  thos<i  called  for  In  the  instructional  objec- 
tives. 

*6  2.  She  should  have  required  at  least  two  prototype  test  Items  along  with 
the  instructional  objectives. 

*B  3.  Each  objective  should  have  been  represented  equally  on  the  criterion 
test. 

*A  4.  Frieda  field-tested  her  first  version  of  the  product  on  too  many 
1  earners . 

*A  5.  She  should  have  pre-tested  the  subjects. 

*B  6.  Too  much  time  (m1d-0ec$ntf>er  to  early  f-ebruary)  was  taken  to  complete 
the  first  version  of  the  product. 

*B  7.  Five  objectives  are  too  many  for  such  a  program. 

*B  8.  More  than  four  colleagues  should  have  reacted  to  the  first  version  of 
the  program. 

*A  9.  Frieda  should  probably  have  been  concerned  that 'the  mean  score  was  21.2. 


Exercise  Two 

Nr.  Smith  has  been  assigned  the  task  of  developing  a  one-week  (approxi- 
mately five  SOnninute  periods)  group-paced  instructional  program  designed  to 
teach  high  school  chemistry  students  to  treat  correctly  certain  analytic  equa- 
tions and  problems  involving  unknown  chemical  elements.    Mr.  Smith  does  not 
attend  to  the  formulation  process  because  this  has  been  done  by  others.  His 
responsibility  is  to  develop  the  actual  instructional  material.    The  instruc- 
tional objective  which  has  been  given  to  him  by  members  of  the  formulation  team 
is  the  following: 


*  Correct  response 
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At  the  conclusion  of  the  instruction  at  least  80  percent  of 
the  learners  will  be  able  to  solve  seven  of  ten  equation 
problems  involving  an  unknown  compound. 

Mr.  Smith  arranges  to  talk  to  several  high  school  chemistry  instruc- 
tors and  a  half  a  dozen  high  school  chemistry  students  to  secure  some  ideas 
as  to  the  proper  tactics  to  employ  in  teaching  the  particular  subject.  The 
students  are  asked  how  much  they  already  know  of  the  topic  so  that  Mr.  Smith 
can  Identify  the  competencies  he  can  build  upon  when  preparing  the  instruc- 
tional sequence. 

He  develops  four  programs,  each  lasting  approximately  40  minutes,  with 
the  expectation  that  these  early  versions  will  be  revised  and  augmented.  He 
adopts  a  "lean"  strategy  in  programming  in  which  he  offers  the  minimum  amount 
of  instructional  materials  that  he  believes  is  requisite,  anticipating  that 
if  the  program  fails  it  will  be  easier  to  add  to  it  than  to  subtract  super- 
fluous material  from  an  effective  program  sequence.    All  of  the  programmed 
material  is  transferred  to  an  audiotape  so  that  it  can  be  coordinated  with 
visual  materials  which  are  presented  on  2  x  2  slides.    He  ultimately  plans 
to  transfer  the  visual  sections  to  a  filmstrip,  but  believes  the  slide  pre- 
sentation will  offer  more  flexibility  for  the  subsequent  addition  or  deletion 
of  modified  visuals. 

He  next  develops  a  criterion  test  consisting  of  thirty  Items  in  which 
the  student  is  presented  with  verbal  descriptions  of  chemical  interactions 
and  asked  to  describe  with  chemical  equations  the  nature  of  the  quantitative 
equations  which  have  been  verbally  described. 

He  tries  the  program  with  four  learners  who  answer,  respectively,  21, 
25,  26,  and  27  Items  correctly  on  the  30-item  test. 

10-18   For   items  10  through  18,  mark  A  if  the  statement  is  correct;  mark  B 
if  It  is  Incorrect. 

*A  10.  Mr.  Smith's  expectation  that  the  early  version  of  the  program  will  be 
revised  is  realistic. 

^  11.  His  students  did  not  perform  as  well  as  they  should  have  on  his  first 
draft  materials. 

^  12.  A  "lean"  prograimlng  strategy  has  previously  been  demonstrated  to  be 
Ineffectual  in  this  type  of  task.  ^ 

*A  13.  Mr.  Smith  should  have  prepared  his  criterion  test  prior  to  the  develop 
ment  of  his  first  version  Instructional  product. 

^  14.  The  use  of  audiotape  and  filmstrip  Is  Inconsistent  with  the  notion  of 
group-paced  programs. 

*A  15.  Mr.  Smith's  criterion  test  was  not  appropriate  for  the  instructional 
objectives  he  had  been  givnn. 

*B  16.  Mr.  Smith,  or  any  similar  programmer,  has  no  responsibility  for 
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evaluating  the  adequacy  of  the  formulation  operation  because  of  the 
late  point  at  which  he  was  Introduced  to  the  development  process. 

*B  17.  Mr.  Smith  should  not  have  consulted  teachers  and  pupils  prior  to  the 
development  of  his  first  draft  of  Instructional  materials. 

^  18.  Mr.  Smith  should  have  developed  first  draft  materials  which  were 
exactly  as  long  as  those  meant  for  the  program. 


19.      Mhlch  one  of  the  following  Is  LEAST  defensible  as  a  component  of  an 
Instructional  program  designed  to  produce  mastery  on  the  objective: 
"To  construct  an  equilateral  triangle,  given  a  straight  edge  and 
compass"? 

A.  Checks  whether  the  learnm-  has  the  prerequisite  knowledge  of 
what  a  triangle  Is  by  having  the  learner  Identify  a  triangle 
given  pictures  of  triangles,  rectangles,  and  other  geometric 
shapes. 

B.  Provides  the  definition  that  an  equilateral  triangle  Is  a  triangle 
in  which  all  sides  and  angles  are  equal. 

*  C.    Asks  the  learner  to  name  an  object  that  has  equal  sides  and  angles. 

D.    Provides  practice  in  the  correct  use  of  a  compass. 


20.      Which  one  of  the  following  objectives  depends  most  heavily  on  a  higher 
level  cognitive  process. 

A.  Exhibits  tolerance  for  others  by  displaying  good  manners  toward 
those  of  minority  groups. 

B.  Correctly  recites  the  Gettysburg  Address  from  memory. 

C.  Displays  interest  in  higher  mathematics  by  volitionally  attending 
lectures  on  this  topic. 

D.  Plays  table  tennis  according  to  rules  well  enough  to  beat  three 
Inexperienced  girls  10X  of  the  time. 

*  E.    Is  able  to  choose  the  best  of  two  solutions  to  a  geometry  probleih 
using  standards  given  by  the  teacher. 
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21.      In  the  following  list.  Identify  the  one  objective  which  makes  most 
clear  what  would  count  as  evidence  that  the  objective  has  been  met. 


A. 

The  student  w1 1 1 

know  six  verbs. 

B. 

The  student  w1 1 1 

learn  the  names  of  the  connon  tools  In  wood  shop. 

C. 

The  student  will 

prefer  cooking  to  sewing. 

♦  0. 

The  student  will 

be  able  to  thread  correctly  a  sewing  machine. 

E. 

The  student  will  pay  attention  as  the  teacher  demonstrates  the 
use  of  the  lathe. 

22-23  Questions  22  and  23  refer  to  the  following  paragraph: 

An  Important  skill  of  the  product  developer  Is  the  ability  to  write 
test  Items  which  match  objectives.    Such  components  of  Items  as 
directions  to  the  learner,  content  dealt  with  In  the  Item,  and  response 
called  for  on  the  part  of  the  learner  should  be  consistent  with  the 
Instructional  objective.    For  questions  22  and  23  below.  Indicate 
whether  or  not  the  objective  and  Item  match. 

22.      Objective;   The  student  will  write  the  course  of  action  most  consistent 

with  the  tenets  of  good  citizenship  outlined  In  class  when 
given  a  social  problem  not  previously  encountered. 

Item:         Choose  a  social  problem  you  are  familiar  with  and  In  less 
than  300  written  words  describe  how  you  would  deal  with 
It  In  terms  of  the  citizen  concepts  described  In  class. 


The  objective  and  Item 
A.  Match 
*  B.   Do  not  match 
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23.      Objective;   The  student  will  be  able  to  arrange  in  chronological 

order  sets  of  three  historical  events  identified  only 
by  brief  descriptive  phrases. 

Item;  Put  in  the  appropriate  order  of  their  occurrence  the 

three  historical  events  alluded  to  in  the  following 
phrases: 

1.  Undeclared  U.S.  war,  often  referred  to  as  "police  action. 

2.  Establishment  of  post  World  War  II  treaty  organization 
in  Southeast  Asia. 

3.  Final  battle  signaling  end  of  major  French  influence 
in  Indo-China. 

The  objective  and  item 

*  A.  Match 

B.    Do  not  I.,  .tch 


Exercise  Three 

Professor  X  is  helping  with  a  teacher  training  program.    He  has  been 
asked  to  use  films  to  attain  some  of  the  objectives  of  the  program.   He  took 
the  following  series  of  steps: 

1.  He  found  a  black  and  white  film  showing  an  argument  between  two 
boys  in  the  school  cafeteria,  culminating  in  a  fist  fight.  The 
narration  described  the  past  history  of  ill  will  between  these 
two  boys. 

2.  He  took  as  his  lesson  objective,  "How  to  deal  with  aggressive  be- 
havior in  children." 

3.  He  prepared  a  printed  handout  listing  "Causes  of  aggression  in 
children." 

4.  He  showed  the  film  to  his  teacher  trainees. 

5.  He  conducted  a  discussion  of  the  film  and  reviewed  the  printed 
handout. 

6.  He  gave  a  test  asking  the  trainees  to  describe  at  least  3  instances 
of  aggressive  behavior  in  children  they  had  observed. 

Please  answer  the  following  questions  about  the  situation  described  above. 
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24.  Professor  X  should  have 

A.  Selected  a  color  film  Instead  of  a  black  and  white  film. 

B.  Reviewed  the  handout  with  the  trainees  before  he  showed  the  film. 

*  C.   Chosen  his  objective  before  he  chose  his  film. 

D.   Asked  for  at  least  5  instances  on  his  test,  rather  than  3. 

25.  The  major  fault  with  the  test  given  is  that 

A.  It  is  too  short  to  be  reliable. 

B.  It  is  difficult  to  score. 

C.  It  is  unrelated  to  the  film. 

*  D.  It  is  unrelated  to  the  objective. 

26.  The  actions  of  Professor  X  violated  which  important  principle  in 
the  design  of  instruction? 

A.  The  materials  should  be  selected  first. 

*  B.   The  objective  should  be  selected  first. 

C.  The  test  Items  should  be  selected  first. 

» 

D.  The  class  activity  should  be  selected  first. 

27.  In  your  Judgment v  was  Professor  X  least  proficient  In 

*  A.   Overall  planning  of  Instruction. 

B.  Text  construction. 

C.  Preparing  handout  materials. 

D.  Classroom  use  of  films. 
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28.      What  reference  would  you  recommend  Professor  X  should  read? 

*  A.    Mager's  poem  on  "where  am  I  going,  how  will  I  get  there,  and 
how  will  I  know  I've  arrived?" 

B.  Any  good  statistics  book. 

C.  A  book  on  making  films. 

D.  Hilgard's  Theories;  of  Learning. 


29.  Suppose  you  are  developing  some  instructional  materials  to  teach  a 
group  of  illiterate  itinerant  farm  workers  how  to  use  knowledge  of 
the  law  to  stay  out  of  trouble.    What  would  be  the  best  opening 
strategy? 

A.  Explain  how  the  law  protects  us. 

B.  Give  a  description  of  the  basic  organization  of  our  courts. 

C.  Ask  them  what  they  think  "law"  means. 
*  D.    Ask  "Why  is  Joe  in  jail  today?" 

Exercise  Four 

Goals  and  objectives  need  to  be  specified  at  different  levels  for 
different  purposes.    But  there  is  a  question  of  who  should  be  most  influential 
in  setting  goals  at  these  different  levels.    Please  match  types  of  persons 
below  with  the  goal  level  at  which  they  should  be  most  influential. 

30.  *  D    General  goals  of  a  school  A.  Instructional  technologist 

31.  *  C    Objectives  for  a  single  course    B.  School  administrators 

32.  *  B     Kinds  of  programs  designed  C.  Teachers 

to  reach  the  school's  goals 


33.  *  A  The  enabling  objectives 
needed  to  attain  course 
objectives  in  a  single  course 


D.  Conminlty  members 
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Exercise  Five 


Many  different  authors  have  published  "models"  of  instructional 
design.  Some  are  labelled  as  "systems"  models,  and  some  are  not.  The 
following  are  steps  which  ma^y  or  may  not  appear  in  a  "systems"  model. 


1.  Pre-testing  for  entrance  competencies. 

2.  Arranging  for  "branching"  programs. 

3.  Analytical  theory  of  media  selection. 

4.  Statement  of  objectives. 

5.  Elaborate  tests  at  5  or  6  "levels" 
in  the  course. 

6.  Individualized  self-paced  method. 


7.  Set  of  instructional  materials. 

8.  Hierarchical  analysis  of  objec- 
tives. 

9.  Tests  to  assess  attainment  of 
objectives. 

10.  Careful  balance  or  "mix"  of  media. 

11.  Self -evaluation  exercises. 

12.  Emphasis  upon  learning  by  discovery. 


Which  three  of  the  above  steps  in  instructional  design  will  most  often 
appear  in  a  "systems"  model? 

A.  1,5,  12 

*  B.  4,  7,  9 

C*  3  f  6  9  8 

D.  4,  6,  10 
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KEY  TO  COMPETENCIES  MEASURED  IN  EACH  TEST  ITEM 
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Test  of  Educational  Research  and  Evaluation  Competencies:    Form  A* 


Final  Pilot  Test  Item  Number  on  General  Specific 

Item  Nunber         137-Item  Tryout  Test        Competency  Category       Competency  Category 


1 

1 

R 

Rl* 

2 

21 

R 

R9 

3 

22 

R  -  E 

G1 

4 

116 

R  -  E 

Ml,  {m)^ 

5 

117 

R  -  E 

Ml,  (H6) 

6 

118 

R  -  E 

Ml,  (MS) 

7 

119 

R  -  E 

G2 

8 

120 

R  -  E 

PI,  (P7) 

9 

121 

R  -  E 

01 

10 

5 

R,  (R  -  E) 

R3,  (D2) 

11 

6 

R,  (R  -  E) 

R3,  (D2) 

12 

8 

R,  (R  -  E) 

R3,  (D2) 

13 

10 

R,  E 

RS,  E3 

14 

11 

R 

R6 

15 

122 

R  -  E 

M4,  (M6) 

16 

1  08 

R  -  E 

P2 

17 

97 

E 

__c 

18 

109 

R  -  E 

P4,  SI ,  (S5) 

19 

110 

R  -  E 

P3 

20 

111 

R  -  E 

SI,  (P3) 

21 

74 

R.  (R  -  E) 

R9,  (SI) 

22 

75 

R  -  E 

SI 

23 

76 

R  -  E 

M2 

24 

77 

R.  (R  -  E) 

R9,  (SI) 

25 

78 

R  -  E 

SI 

26 

88 

R  -  E 

S5 

27 

26 

R.  R  -  E 

R4,  D3 

foescfiptions  of  the  skills  and  knowledge  are  given  in  the  early  pages  of  Chapter  IV. 
''Parentheses  denote  secondary  emphasis. 

^is  itaa  coald  not  be  classified  wore  specifically  than  as  a  general  evaluation  skill. 

«The  Matching  of  items  with  competencies,  as  shown  In  this  appendix,  is  based  on  logical 
rather  than  eaplrical  grounds.  .  w..^ 
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Test  of  Educational  Research  and  Evaluation  Competencies:   Fonn  A  (continued) 


Final  Pilot  Test  Item  Nunber  on  General  Specific 

Item  Nunber         137-Item  Tiyout  Test        Competency  Category       Conpetency  Category 


28 

30 

K»  R  -  t 

29 

37 

K  -  t 

30 

Z7 

A^ 

31 

R  -  t 

CI  /ec\ 

51,  (S5) 

32 

79 

R  -  t 

51 ,  (S5) 

33 

80 

R  -  E 

SI.  (S5) 

34 

45 

R  "  E 

SI 

35 

57 

R  -  E 

SI 

36 

58 

R  -  E 

P2 

37 

59 

R  -  E 

S4 

38 

42 

R  -  E 

SI 

3^ 

73 

R  -  E 

P3.  (P8) 

40 

13 

R 

R8,  R9 

41 

113 

R  -  E 

P3 

42 

114 

R  -  E 

P5 

43 

65 

R  -  E 

S2 

44 

90 

R  -  E 

P6 

45 

135 

R  -  E 

63 

trough  a  typographical  error,  this  Item  failed  to  be  numbered  on  the  tryout  test. 
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Test  of  Educational  Research  and  Evaluation  Competencies:   Form  B 


i 
i 

i 

f 
I 
I 
I 
i 
I 
I 
I 

•ERIC  I 


not  Test 
1  Nuntfier 

Item  Nuiroer  on 
137-Item  Tryout  Test 

ben  era  i 
Competency  Category 

opec 1  T 1  C 

Conpetency  Category 

1 

3 

R 

R2 

2 

125 

(R-  E) 

(M6) 

^3 

126 

R  -  E 

D3 

4 

127 

R,  (R  -  E) 

R5,  R6,  R7,  (M5) 

5 

128 

R  -  E 

03 

6 

129 

R  -  E 

03 

7 

130 

(R-E) 

(D3) 

8 

23 

R  -  E 

P3 

9 

24 

R 

R6,  (R9) 

10 

134 

R-E 

61 

11 

124 

R-E 

N6 

12 

98 

E 

El 

13 

99 

R  -  E 

62 

14 

100 

E 

E2,  (E3) 

15 

115 

R  -  E 

Ml 

16 

107 

R-E 

P2 

17 

25 

R  -  E 

SI 

18 

131 

R  -  E 

SI,  (S5.  P3) 

19 

132 

R-  E.  (R) 

SI,  (SB,  P3,  R9) 

20 

31 

R,  R  -  E 

R4,  D3 

21 

36 

R-E 

03 

22 

39 

R-E 

03,  (02) 

23 

32 

R  -  E 

04 

24 

33 

R  -  E 

04 

25 

34 

R-  E.  (R) 

05,  (R8) 

26 

new  item 

R-E 

SI 

27 

81 

R  -  E 

N3 

28 

82 

R  -  E 

N3 

29 

83 

R  -  E 

M3,  (S3) 
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Test  of  Educational  Research  and  Evaluation    Competencies:    Form  B  (continued) 


Final  Pilot  Test  Item  Nunber  on  General  Specific 

Item  Nunber         137-Item  Tryout  Test         Competency  Category       Competency  Category 


30 

48 

R  -  E 

SI 

31 

49 

R  -  E 

SI 

32 

50 

R  -  E 

P2 

ft? 

Of 

B  _  F 
l\  ~  c 

34 

12 

R,  R  -  E 

R8,  SI 

35 

92 

R  -  E 

SI,  (S5) 

36 

93 

R  -  E 

SI,  (S5) 

37 

94 

R  -  E 

P6,  (PS) 

38 

64 

R  -  E 

S2 

39 

72 

R  -  E.  (R) 

S4,  (R9) 

40 

66 

R  -  E 

S6 

41 

14 

R 

R9 

42 

15 

R.  E 

R8,  E4 

43 

new  Item 

R  -  E 

G3 

44 

91 

(R  -  E) 

(P6) 

45 

136 

R  -  E 

G3 
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QUESTIONNAIRE  RECIPIENTS  FOR 
RESEARCH  TRAINING  STUDY 
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State  Departments  of  Education 


I 
I 

i 

i 


I 


I 
I 
I 
I 


it 
ic 
ie 


ic 


ic 


ic 
ic 


Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

Fl oriia 

Georgia 

Hawaii 

Idaho 

Illinois 

Indiana 

Iowa 

Kansas 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 


Nevada 

New  Hampshire 
New  Jersey 
New  Mexico 
New  York 
North  Carolina 
North  Dakota 
Ohio 

Oklahoma 
Oregon 
Pennsyl vania 
Puerto  Rico 
Rhode  Island 
South  Carolina 
South  Dakota 
Tennessee 
Texas 
Utah 
Vermont 
Virginia 
Virgin  Islands 
Washington 
West  Virginia 
Wisconsin 
Wyomi  ng 

District  of  Columbia 


*0n  the  basis  of  their  replies  to  the  initial  letter  of 
inquiry,  starred  organizations  were  not  sent  copies  of  the 
questionnaire  for  non-de<jree  grantingTnstitutions,  since  they 
had  indicated  that  they  offer  no  RDDE  training  activities. 
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USOE -Supported  R&D  Centers  and 
Regional  Educational  Laboratories 


R&D  Centers 

Center  for  the  Advanced  Study  of  Educational  Administration,  Eugene 
Center  for  Research  and  Development  in  Higher  Education,  Berkeley 
Center  for  the  Study  of  Evaluation,  Los  Angeles 
Center  for  the  Study  of  Social  Organization  of  Schools,  Baltimore 
Learning  Research  and  Development  Center,  Pittsburgh 
Research  and  Development  Center  for  Teacher  Education,  Austin 
Stanford  Center  for  Research  and  Development  in  Teaching,  Palo  Alto 
Wisconsin  Research  and  Development  Center,  Madison 

Regional  Educational  Laboratories 

Appalachia  Educational  Laboratory,  Inc.,  Charleston,  W.  Virginia 
CEMREL,  Inc.,  St.  Ann,  Missouri 
Center  for  Urban  Education,  New  York 

Far  West  Laboratory  for  Educational  Research  and  Development,  Berkeley 
Mid-continent  Regional  Educational  Laboratory,  Kansas  City,  Missouri 
Northwest  Regional  Educational  Laboratory,  Portland 
National  Laboratory  for  Higher  Education,  Durham,  N.  Carolina 
Research  for  Better  Schools,  Inc.,  Philadelphia 
Southwest  Cooperative  Educational  Laboratory,  Albuquerque 
Southwest  Educational  Development  Laboratory,  Austin 
Southwest  Regional  Laboratory,  Inglewood,  California 
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Regional  Education  and  School  Study  Councils 

Adirondack  Center  for  Research,  Experimentation  and  Services  (ACRES), 
Plattsburgh,  New  York 

Anne  Tyskling  Consortium  for  the  Mid-west,  Chicago,  Illinois 

Anthracite  Institute  for  Developing  Schools,  Wilkes-Barre,  Pennsylvania 

Arkansas  School  Study  Council,  Fayetteville,  Arkansas 

Associated  Public  School  Systems,  New  York,  New  York 

Bay  Area  School  Study  Council,  Palo  Alto,  California 

Bureau  of  Educational  Research,  Urbana,  Illinois 

Bureau  of  Educational  Research  and  Service,  Tucson,  Arizona 

Bureau  of  Educational  Research  and  Services,  Tempe,  Arizona 

Capital  Area  School  Development  Association,  Albany,  New  York 

Catskill  Area  School  Study  Council,  Oneonta,  New  York 

Central  Illinois  School  Development  Council,  Normal,  Illinois 

Central  New  York  School  Study  Council,  Syracuse,  New  York 

Central  Ohio  Educational  Research  Council,  Columbus,  Ohio 

Central  School  Boards  Committee  for  Educational  Research,  New  York,  New  York 

Colgate  School  Study  Council,  Hamilton,  New  York 

Connecticut  Association  for  the  Advancement  of  School  Administration,  Inc., 
Hartford,  Connecticut 

Cooperative  School  Service  Center,  Amherst,  Massachusetts 

Cooperative  School  Study  Council,  Sacramento^  California 

Division  of  Research  and  Guidance.  Los  Angeles,  California 

East  Central  Indiana  School  Study  Council  Joint  Program,  Muncie,  Indiana 

East  Texas  School  Study  Council ,  Commerce,  Texas 
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Educational  Administration  Center,  Dekalb,  Illinois 

Educational  Improvement  Center,  Glassboro,  New  Jersey 

Educational  Research  arid  Development  Council,  Minneapolis,  Minnesota 

Educational  Research  and  Development  Council  of  Delaware,  Inc.,  Newark, 
Delaware 

Educational  Research  and  Service  Bureau,  Philadelphia,  Pennsylvania 

Educational  Research  Council  of  Anerica,  Cleveland,  Ohio 

Education  Council  for  School  Research  and  Development,  Mineola,  New  York 

ERDC  of  Central  Minnesota,  St.  Cloud,  Minnesota 

ERDC  of  Northeast  Minnesota  (RAND),  Duluth,  Minnesota 

ERDC  of  Northwest  Minnesota,  Red  Lake  Falls,  Minnesota 

ERDC  of  Southwest  and  West  Central  Minnesota,  Marshall,  Minnesota 

Florida  Educational  Research  and  Development  Council,  Inc.,  Gainesville, 
Florida 

Fox  Valley  Curriculum  Study  Council,  Oshkosh,  Wisconsin 
Genesee  Valley  School  Development  Association,  Rochester,  New  York 
Gulf  School  Research  Development  Association,  Houston,  Texas 
Idaho  State  University,  Pocatello,  Idaho 

Indiana  Public  School  Study  Council  Joint  Program,  Muncle,  Indiana 

Indiana  State  University  Educational  Development  Council,  Terre  Haute, 
Indiana 

Institute  for  Educational  Research,  Downers  Grove,  Illinois 
Iowa  Center  for  Research  In  School  Administration,  Iowa  City,  Iowa 
Kansas  City  Metropolitan  School  Study  Group,  Kansas  City,  Missouri. 
Lakeshore  Curriculum  Study  Council,  Milwaukee,  Wisconsin 
Lehigh  Valley  School  Study  Council,  Bethlehem,  Pennsylvania 
Metropolitan  Detroit  Bureau  of  School  Studies,  Detroit,  Michigan 
Metropolitan  School  Study  Council,  New  York,  New  York 
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Michigan  Educational  Research  Association,  East  Lansing,  Michigan 

Mid-Hudson  School  Study  Council,  New  Paltz,,  New  York 

Mid-Monmouth  Educational  Council,  Long  Branch,  New  Jersey 

New  England  School  Development  Council,  Newt.on,  Massachusetts 

New  Jersey  School  Development  Council  ,'  New  Brunswick,  New  Jersey 

New  Jersey  Urban  Schools  Development  Council,  Trenton,  New  Jersey 

New  Mexico  Research  and  Study  Council,  Albuquerque,  New  Mexico 

North  Country  School  Study  Council,  Potsdam,  New  York 

Northeastern  Indiana  School  Study  Council  Joint  Program,  Muncie,  Indiana 

Northern  Illinois  Cooperative  in  Education  (NICE),  Dekalb,  Illinois 

Northwest  Indiana  Public  School  Study  Council  Joint  Program,  Muncie, 
Indiana 

Oklahoma  Public  School  Research  Council,  Stillwater,  Oklahoma 
Oregon  School  Study  Council,  Eugene,  Oregon 
Oswego  Area  Council  for  Educational  Development,  Oswego,  New  York 
Peninsula  Study  Council,  Palo  Alto,  California 

Pennsylvania  School  Study  Council,  Inc.,  University  Park,  Pennsylvania 
Philadelphia  Suburban  School  Study  Council,  Philadelphia,  Pennsylvania 
Public  Schools  for  Cooperative  Research,  Knoxville,  Tennessee 
Rocky  Mountain  School  Study  Council ,  Denver,  Colorado 
School  Information  and  Research  Service,  Seattle,  Washington 
SCOPE,  Long  Island,  New  York 

Southeast  Alabama  Superintendents  and  Principals  Conference,  Troy,  Alabama 

Southern  Minnesota  Educational  Research  and  Development  Council,  Rochester, 
Minnesota 

Southern  Tier  School  Study  Council ,  Alfred,  New  York 
South  Florida  Educational  Planning  Council,  Tampa,  Florida 
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S.  W.  New  York  Association  for  the  Improvement  of  Instruction,  Fredonla^ 
New  York 

Southwestern  Ohio  Educational  Research  Council,  Inc.,  Middletown,  Ohio 

Tri-State  Area  School  Study  r^uncll,  Pittsburgh,  Pennsylvania 

University  of  Virginia  COiincll  for  Educational  Research,  Charlottesville 
Virginia 

Upper  Wabash  Valley  School  Study  Council  Joint  Program,  Muncle,  Indiana 

Washington  Area  School  Study  Council,  La  Plata,  Maryland 

Western  Educational  Council  of  South  Dakota,  Ellsworth  Air  Force  Base, 
South  Dakota 

Western  New  York  School  Development  Council,  Buffalo,  New  York 
West  Texas  School  Study  Council,  Lubbock,  Texas 
Wyoming  School  Study  Council,  Laramie,  Wyoming 
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Professional  Associations 


51 


American  College  Testing  Program,  Inc.,  Iowa  City,  Iowa 

*  American  Council  on  Education,  Washington,  D.  C. 

*  American  Personnel  and  Guidance  Association,  Washington,  D.  C. 

*  American  Psychological  Association,  Washington,  D.  C. 

*  American  Sociological  Association,  Washington,  D.  C. 

Association  for  Supervision  and  Curriculum  Development,  Washington,  D.  C. 
College  Entrance  Examination  Board,  New  York,  New  York 


i  *  National  Acadeiny  of  Sciences,  National  Research  Council,  Washington, 

^  D.  C. 


*  National  Council  on  Measurement  in  Education,  East  Lansing,  Michigan 

*  National  Society  for  the  Study  of  Education,  Chicago,  Illinois 

*  Union  for  Research  and  Experimentation  in  Higher  Education,  Yellow 

Springs,  Ohio 


I 


*0n  the  basis  of  their  replies  to  the  Initial  letter  of  Inquiry, 
starred  organizations  were  not  sent  copies  of  the  questionnaire  for  non- 
degree  granting  Institutions,  since  they  had  Indicated  that  they  offer 
no  RDDE  training  activities. 
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Federal  Agencies  and  Offices 

Center  fbr  Urban  Education,  New  York 

Division  of  State  Agency  Cooperation,  USOE 

Educational  Conmunl cation  Office,  USOE 

Educational  Leadership  Branch,  Bureau  of  Education  Personnel 
Development,  USOE 

Educational  Staff  Seminar,  Washington,  D.  C. 

Employee  Development  Program,  USOE 

TTT  Program,  Bureau  of  Education  Personnel  Development,  USOE 
Washington  Internships  In  Education,  Washington,  D.  C. 
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Private  Research  and  Development  Organizations 


Acadeiny  for  Educational  Development,  Inc.,  New  York,  New  York 

American  Institutes  for  Research,  Palo  AUo,  California 

Associates  for  Research,  Inc.,  Philadelphia,  Pennsylvania 

Behavioral  and  Social  Sciences  Laboratory,   Philadelphia,  Pennsylvania 

Borg-Warner  Educational  Systems,  Niles,  Illinois 

Columbia  Scientific  Industries  Corporation,  Austin,  Texas 

Education  and  Training  Consultants  Co.,  Los  Angeles,  California 

Education  and  World  Affairs,  New  York,  New  York 

Educational  Development  Corporation,  Palo  Alto,  California 

Educational  Materials  and  Equipment  Company,  Bronxville,  New  York 

Educational  Testing  Service,  Princeton,  New  Jersey 

Gal  ton  Institute,  Beverly  Hills,  California 

General  Systems  Industries,  Inc. ,  Torrance,  California 

Gould  Scientific,  Washington,  D.  C. 

Human  Resources  Research  Organization,  Alexandria,  Virginia 

institute  for  Community  Studies,  Kansas  City,  Missouri 

KDI  Corporation,  Forera  Corporation,  Washington,  D.  C. 

KMS  Industries,  Inc.,  Electronic  Futures  Division,  North  Haven,  Connecticut 

Lester  Associates,  Inc.,  Thornwood,  New  York 

Life  Sciences,  Inc.,  Fort  Worth,  Texas 

Thomas  A.  Edison  Industries  Laboratory,  West. Orange,  New  Jersey 

The  Mitre  Corporation,  Bedford  Operations,  Bedford,  Massachusetts 

New  England  Research,  Inc.,  Worchester,  Massachusetts 

Palo  Alto  Medical  Research  Foundation,  Palo  Alto,  California 

Panoramic  Studios,  Philadelphia,  Pennsylvania 

Parco  Scientific  Company,  Warren,  Ohio 

Performance  Research  Incorporated,  Washington,  D.  C. 


*0n  the  basis  of  their  replies  to  the  Initial  letter  of  inquiry, 
starred  organizations  were  not  sent  copies  of  the  questionnaire  for  non- 
degree  granting  institutions,  since  they  had  indicated  that  they  offer 
no  RDDE  training  activities. 
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Planning  Research  Corporation,  Los  Angeles,  California 
Psychological  Research  Service,  Inc.,  Austin,  Texas 
Rand  Corporation,  Santa  Monica,  California 
Rheem  Manufacturing  Comparty,  Los  Angeles,  California 
Science  Research  Associates,  Chicago,  Illinois 

Scott  Paper  Comparty,  S.  B.  Sutpin  Research  Center,  South  Hadley,  Mass. 
Simulaids,  Inc.,  Woodstock,  New  York 

The  Singer  Company,  Link  Division  -  Education  Systems,  Binghamton,  N.Y. 
Stanford  Research  Institute,  Menlo  Park,  California 
Synectics,  Inc.  Cambridge,  Massachusetts 

Syracuse  University  Research  Corporation,  Policy  Institute,  Syracuse,  N.Y. 
Systems  Development  Corporation,  Santa  Monica,  California 
Tracor,  Inc.,  San  Diego  Laboratory,  San  Diego,  California 
Viewlex,  Inc.,  Holbrook,  New  York 

Westinghouse  Electric  Corporation:  ^ 
Behavioral  Systems  Division,  New  York,  New  York 
Learning  Aids  Division,  Pittsburgh,  Pennsylvania 
Measurement  Research  Center,  Iowa  City,  Iowa 
■  New  England  Educational  Data  Systems,  Waltham,  Massachusetts 
Project  Annapolis  Division,  Annapolis,  Maryland 
Project  Plan  Division,  Palo  Alto,  California 


*0n  the  basis  of  their  replies  to  the  initial  letter  of  inquiry, 
starred  organizations  were  not  sent  copies  of  the  questionnaire  fbr  non 
degree  granting  institutions,  since  they  had  indicated  that  they  offer 
no  RODE  training  activities. 
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APPENDIX  M 


QUESTIONNAIRES  AND  COVER  LEHERS 
FOR  RESEARCH  TRAINING  SURVEY 


COVER  LETTER  FOR  Q 


M.3 

ameRican  eiJiK:i  \  i  lai  mm 
ReseaRCH  associarion 

1 


Dear  Colleague: 

The  American  Educational  Research  Association  Is  currently  conducting 
a  USOE-supported  stu4y  to  Identify  and  describe  all  formal  training 
programs  In  educational  research  and  research-related  areas  In  the 
United  States.   From  the  results  of  the  study  we  will  report  to  USOE 
on  the  current  status  of  research  training  and  recommend  actions  to 
continue  and  Improve  training  opportunities  for  educational  researchers. 

As  part  of  this  effort  we  are  distributing  the  enclosed  questionnaire 
to  key  persons  In  master's  and  doctoral  degree  granting  Institutions. 
Specifically,  we  would  like  to  ask  you  to  answer  a  single  question  on 
the  enclosed  questionnaire  and  then,  depending  on  your  answer,  either 
return  It  to  us  or  forward  It  to  the  appropriate  person  In  your 
Institution. 

We  would  be  most  grateful  If  you  would  take  a  few  minutes  to  read  the 
definitions  at  the  beginning  of  the  questionnaire.    If,  on  the  basis 
of  those  definitions,  you  determine  that  your  Institution  does  not 
offer  a  graduate  program  In  educational  research  and  research-related 
areas,  simply  check  Item  1  and  return  the  questionnaire  In  the 
enclosed  envelope. 

If  you  do  have  a  graduate  educational  research  or  research- related 
program  according  to  our  definitions,  please  forward  the  questionnaire 
and  this  cover  letter  to  the  director  of  the  program  or  to  the  faculty 
member  who  Is  best  able  to  describe  the  program.    He  should  then 
complete  the  questionnaire.  Including  the  Information. In  Item  11 
concerning  his  position,  and  return  the  questionnaire  to  us. 

Your  response  Is  most  Important  to  our  study;  In  order  to  meet  our 
contract  deadline,  we  would  be  most  appreciative  If  the  questionnaire 
could  be  retunied  to  us  no  later  than  August  9.    If  you  have  questions 
concerning  the  studly,  you  may  contact  me  at  the  address  Indicated  on 
the  return  envelope. 

Sincerely  yours, 

Blaine  R.  Worthen,  Chairman 
Task  Force  on  Research  Training 


1126  SIXTEENTH  STREET,  N.W.    •  WASHINGTON,  O.  C.  20036  •  202/2?3-948S 

"  516 


M.5 


Q':    QUESTIONNAIRE  FOR  DEGREE  GRANTING  INSTITUTIONS 


A  ERA  Research  Training  Study 


In  this  questionnaire,  educational  research  and  research -related  activities  are  defined  to  Include 
(1)  educational  research.  (2)  the  design  and  development  of  Instructional  Products  and  programs, 
(3)  the  dissemination  of  products  and  curricula  to  the  educational  community,  and  (4)  the  evaluation 
of  materials  and  programs,  both  as  part  of  the  deve1opment..effort  and  as  an  ongoing  monitoring  process. 

Educational  researchers  and  research-related  personnel  thus  include  developers  and  disseminators  as 
wen  as  research  methodologists.  statisticians,  and  experts  in  measurement  and  evaluation.   They  also 
Include  researchers  in  substantive  fields,  such  as  science  education.   (Note,  however,  that  such 
persons  are  specialists  in  research  in  science  education,  not  sped alistrirT  science  education  per  sej 

Given  the  above  definitions,  do  you  have  a  graduate  program  for  the  preparation  of  educational 
researchers  and/or  researdi-related  personnel?   That  is,  do  you  offer  an  identifiable  field  of 
specializatjon  in  educational  research,  development,  diffusion  or  evaluation— one  whicfinFequlvalent 
to  other  areas  of  specialization  in  the  department,  school  or  college  of  education? 

(Please  check  CNE)  YES|    |  NOQ 

If  you  answered  NO  ^bove,  please  provide  us  with  your  name,  position  and  Institution  in  the  space 
below  and  return  the  questionnaire  to  us  in  the  accompanying  envelope. 


If  you  answered  YES  above,  please  forward  the  questionnaire  and  cover  letter  to  the  program  director 
or  other  person  and  ask  him  to  complete  items  2  through  11;  he  may  then  return  the  questionnaire  to 
us  in  the  envelope  provided. 


iSJ!!^'?'*^''^^'*  ^"^^  QUESTIONNAIRE,  THE  PROGRAM  FOR  EDUCATIONAL  RESEARCHERS  AND  RESEAaCH-RELATED 
PERSONNEL  WILL  BE  RCFfiRRED  TO  SIMPLY  AS  THE  EDUCATIONAL  RESEARCH  PROGRAM. 

2  How  long  has  your  educotional  research  program  been  in  existence?   (yrs.) 

3  What  degrees  are  offeied  in  the  educational  research  program?  (Please  check) 

MA  MS   MEd  EdO  PhD  Other  (specify)   


How  many  degree  candidates  vere  specializing  in  educational  research  at  the  end  of  the  1970-71 
academic  year? 

Master's  level:    full  time   part  time  

Doctoral  level:    full  time    part  time  


Please  indicate  which  of  the  following  criteria  you  consider  in  determining  admission  to  your 
educational  research  program.   (Please  check  one  column  for  each  item.) 


Primary    Secondary  No 
Importance  Importance  Importance 


GRE  Total  Score  (min.  acceptable  ) 

GRE  V'^bal  (min.  acceptable  ) 

GRE  Quantitative  (niin.-^acceptable   ) 

Miller  Analogies  Test  (min.  acceptable  _ 

Other  standardized  test 

(Please  specify  


Underaraduate  grade  point  average 

(min.   on  a  scale  of        )  ' 

Graduate  grade  point  average 

(min.  on  a  scale  of  ) 

Recomnendatl ons  of  Instructors /emol overs 


the  dtssemination  of  products  and  curricula  to  the  educational  community,  and  (4)  the  evaluation 
of  materials  and  programs,  both  as  part  of  the  development  effort  and  as  an  ongoing  monitoring  process. 

Educational  researchers  and  research-related  personnel  thus  Include  developers  and  disseminators  as 
well  as  research  methodologlsts,  statisticians,  and  experts  in  measurement  and  evaluation.   They  also 
Include  researchers  in  substantive  fields,  such  as  science  education.   (Note,  however,  that  such 
persons  are  specialists  in  research  in  science  education,  not  sped alistsTni  science  education  fier  se.) 

Given  the  above  definitions,  do  you  have  a  graduate  program  for  the  preparation  of  educational 
researchers  and/or  researdi -related  personnel?   That  is,  do  you  offer  an  identifiable  field  of 
specialization  in  educational  research,  development,  diffusion  or  evaluation -one  whidnTequivalent 
to  other  areas  of  specialization  in  the  department,  school  or  college  of  education? 

(Please  check  CNE)  YE$|    |  NOQ 

If  you  answered  NO  ^bove,  please  provide  us  with  your  name,  position  and  institution  in  the  space 
below  and  return  the  questionnaire  to  us  in  the  accompanying  envelope. 


If  you  answered  YES  above,  please  forward  the  questionnaire  and  cover  letter  to  the  program  director 
or  other  person  an?  ask  him  to  complete  items  2  through  11;  he  may  then  return  the  questionnaire  to 
us  in  the  envelope  provided* 


IN  THE  REMAINDER  OF  THIS  QUESTIONNAIRE.  THE  PROGRAM  FOR  EDUCATIONAL  RESEARCHERS  AND  RESEAROi-RELATED 
PERSONNEL  WILL  BE  RCFLRRED  TO  SIMPLY  AS  THE  EDUCATIONAL  RESEARCH  PROG^  KtbtARW  RELATED 

2  How  long  has  your  educntional  research  program  been  in  existence?  (yrs.) 

3  What  degrees  are  offered  in  the  educational  research  program?  (Please  check) 

MA  MS   MEd  EdO  PhD  Other  (specify)   


How  many  degree  cand1d:»Us  /.ere  specializing  in  educational  research  at  the  end  of  the  1970-71 
academic  year? 

Master's  level:    full  time    part  time  

Doctoral  level;    full  time    part  time  

Please  Indicate  which  of  the  following  criteria  you  consider  in  determining  admission  to  your 
educational  research  program.    (Please  check  one  column  for  each  item.) 


6RE  Total  Score  (roin.  acceptable  ) 

6RE  Verbal  (min.  acceptable  ) 

6RE  Quantitative  (n.in,  acceptable  ) 

Miller  Analogies  Test  (min.  acceptable  ^ 

Other  standardized  test 

(Please  specify  


Primary    Secondary  No 
Importance  Importance  Importance 


Undergraduate-  grade  point  average 
(min.  on  a  scale  of 


Graduate  grade  point  average 

(nrtn.  on  a  scale  of  ) 

Recommendations  of  instructors/employers 

Previous  course  work  in  particular  araas         Ol  / 
(Please- specify  areas  • 

Previous  research  experience 

Previous  school  experience 

Expressed  Interest  In  educational  research 

Undergraduate  major 

(Please  specify  ) 


Age  (Please  specify  j 

Interview 

Other  (Please  specify   ) 


M.6  ^ 

Please  describe  the  minimum  registration  requirements  for  part  time  students  In  your  program. 

 credit  hours  per  academic  term  (Give  nurrtber  of  hours) 

courses  per  academic  term  (Give  number  of  courses) 

Academic  term  (Please  check  ONE):   semester   quarter 

Is  a  minimum  period  of  full  time  residence  required  before  the  awarding  of  a  degree?  YESi    I  NOl  I 
If  YES.  what  1r»  the  length  of  that  period?   

Please  Indicate  which  of  the  following  best  describes  the  primary  focus  of  your  educational  research 
program,  keeping  In  mind  that  your  response  will  Indicate  the  major  areas  In  which  researchers  and 
research -rel ate d  personnel  are  prepared.  (It  may  be  necessary  to  specify  more  than  one  area  or 
empnasis»  out  please  limit  your  choices  to  those  which  describe  the  major  orientation  of  the  program.) 
(Please  check  the  appropriate  Items.) 

  Research  methodology/deslgn/stati sties 

  Measurement  . 

  Evaluation 

  Adhilnl  strati  on 

  Curriculum  and  Instruction 

  Education  subject  areas   (Please  specify  ) 

  Counseling  and  ct<1  dance 

  Psychology 

  Vocational  educatlim 

  Special  education 

  History  and  philosophy  of  education 


  Educational  product  or  program  development 

  Dissemination  and  diffusion  of  educational  materials  and  programs 

  Other   (Please  specify  ) 

For  each  of  the  areas  below»  Indicate  how  many  courses  are  usually  regul red  as  part  of  the  educational 
research  program. 


Required  for         Required  for 
Master's  Doctorate 
(If  applicable)      (If  appllcAle) 


Introduction  to  educational  research 
Advanced  research  methods 
Statistics 

Computer  methods  and  use 
Testing  (use  of  standardized  tests) 
Measurement  and  test  construction 
Evaluation 

Curriculum  and  instruction 
Educational  administration 
Research  management 
Communications 
Marketing 

Politics  or  finance  of  education 

Educational  change  and  change  strategies 

Product  development 

Instructional  sequencing 

Psychology 

Sociology 

History/philosophy  of  education 

Other  (Please  specify  ) 


10    Mhlch  one  or  more  of  the  following  types  of  experience  are  part  of  your  educational  research  program? 
(Please  check  the  appropriate  column.) 


Required  Available 


Asslstantship  or  Internship  within  your  program 

(a)  Assistance  across  several  projects  or  programs 

(b)  Assignment  to  a  particular  project  or  program 
Asslstantship  or  Internship  In  a  campus  research  unit  outside 

your  own  program 

(a)  Assistance  across  several  projects  or  programs 

(b)  Assignment  to  a  particular  project  or  program 
Asslstantship  or  Internship  In  an  off-campus  unit 

(Please  describe     .  _  ) 

Apprenti ceslilp ;;to >a  senior  researcher*  but  not  for  a  specific 
jjroJec^Miro^^ 


^     ,     ,     ,  ,   UIW NlliaP'^  ll'im  Bf  WiiV  gflbditlbnal  reseurcir 

program,  keeping  In  mind  that  your  response  will  indicate  the  major  areas  In  which  researchers  and 
nssearch-related  personnel  are  prepared,   (It  may  l>e  necessary  to  specify  more  than  one  area  of — 
empnasiSg  out  please  limit  your  choices  to  those  wSiidi  describe  the  major  orientation  of  the  program.) 
(Please  check  the  appropriate  Items.)  —  r  i 

  Research  methodology/design/statistics 

■  Measurement 

  Evaluation 

Adnlnlstratlon 
  Curriculum  and  Instruction 

  Education  subject  areas   (Please  specify  ) 

  Counseling  and  guidance  ' 

  Psychology 

  Vocational  education 

  Special  education 

  History  and  philosophy  of  education 


Educational  product  or  program  development 
Dissemination  and  diffusion  of  educational  materials  and  programs 
Other  (Please  specify   


For  each  of  the  areas  below,  indicate  how  man 
research  program. 


hoji  many  courses  are  usually  required  as  part  of  the  educational 


Required  for 

Master's 
(If  applicable) 


Required  for 
Doctorate 
(If  applicable) 


Introduction  to  educational  research 
Advanced  research  methods 
Statistics 

Computer  methods  and  use 
Testing  (use  of  standardized  tests) 
Measurement  and  test  construction 
Evaluation 

Curriculum  and  instruction 
Educational  adknlnistratlon 
Research  management 
Communications 
Marketing 

Politics  or  finance  of  education 

Educational  change  and  change  strategies 

Product  development 

Instructional  sequencing 

Psychology 

Sociology 

Hi  story/phi  lot^ophy  of  education 

Other  (Please  specify   


J 


10    Which  one^or  more  of  the  following  types  of  experience  are  part  of  your  educational  research  program? 
(Please  check  the  appropriate  column,)  ^ 


Required  Available 


Assistantship  or  internship  within  your  program 

(a)  Assistance  across  several  projects  or  programs 

(b)  Assignment  to  a  particular  project  or  program 
Assistantship  or  internship  In  a  campus  research  unit  outside 

your  own  program 

(a)  Assistance  across  several  projects  or  prograais 

(b)  Assignment  to  a  particular  project  or  program 
Assistantship  or  internship  In  an  off-campus  unit 

(Please  describe    ) 

Apprenticeship  to  a  senior  researcher,  but  not  for  a  specific 

project  Of  projects 
Technical  consultant  services  for  faculty t  students  and 

administrators 
Other  research  experience 

(Please  describe   ) 


11     TO  BE  COMPLETED  BY  ALL  RESPONDENTS: 


and  Position 


Department  and  Institution 
Field  of  specialization 


M  7 

LETTER  OF  INQUIRY  TO  SELECTED  NO^I-DEGREE  GRANTING  ORGANIZATIONS 

UNIVERSITY  OF  COLORADO 
BOULOeR.  COLORADO  803O2 


UAMOMATOMV  OP  KOUCATtONAU  MKSKAMCH 


Dear  Sir: 

The  American  Educational  Research  Association's  Task  Force  on  Research 
Training  is  currently  engaged  In  a  study  of  the  educational  community's 
need  for  personnel  to  conduct  research  and  research-related  activities 
in  education.    From  the  results  of  this  study  the  Association  intends 
to  make  recommendations  to  the  U.  S.  Office  of  Education  for  the  train- 
ing and  retraining  of  research  personnel. 

As  part  of  this  endeavor  we  need  to  know  what  formal  opportunities  exist 
for  the  training  of  educational  researchers  and  related  personnel.  Most 
such  opportunities  will  be  found  in  college  and  university  departments 
of  education.    It  has  comi^  to  our  attention,  however,  that  other  institu- 
tions and  organizations  —  among  them  the  state  education  departments  — 
often  provide  somewhat  different  kinds  of  training  In  the  form  of 
institutes  and  workshops  on  research  topics.    There  may  also  be  more 
extensive,  though  still  short-term,  training  of  personnel  for  particular 
research-related  projects  (for  example,  the  evaluation  of  programs  con- 
ducted under  Titles  I  and  III  of  the  Elementary  and  Secondary  Education 
Act  of  1965).    In  addition  seminars  are  sometimes  held  specifically  to 
disseminate  information  on  new  research  results  or  the  application  of  new 
techniques. 

We  regard  all  of  these  as  Important  elernents  in  the  training  of  educational 
researchers  and  wish  to  include  them,  along  with  the  degree-granting 
programs.  In  our  overall  description  of  training  opportunities.    For  this 
reason  we  are  asking  you  to  provide  us  with  a  list  of  research  and  research 
related  training  activities  which  are  supported  or  administered  by  the 
department  of  education  for  your  state.    We  will  also  need  to  know  the 
name  of  a  person  to  contact  for  more  detailed  information  on  each  training 
program. 

In  order  to  meet  our  project  deadline,  we  would  like  to  receive  the 
preliminary  information  by  June  7.   We  greatly  appreciate  any  help  you 
are  able  to  give  us. 

Yours  sincerely. 
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ameRican  eoucaTionaL 
ReseaRCH  associaiion 


COVER  LEHER  FOR  Q' 


Dear  Colleague: 

The  American  Educational  Research  Association  Is  currently  conducting 
a  USQE-supported  study  to  Identify  and  describe  all  formal  training 
programs  In  educational  research  and  research-related  areas  In  the 
United  States.    From  the  results  of  the  study  we  will  report  to  USOE 
on  the  current  status  of  research  training  and  recommend  actions  to 
continue  and  Improve  training  opportunities  for  educational  research 
and  research-rel  ated  personnel . 

As  part  of  this  effort,  we  are  distributing  the  enclosed  questionnaire 
to  key  persons  within  those  agencies  and  organizations  concerned  with 
educational  research  and  research-rel  ated  activities.  Specifically, 
we  would  like  to  ask  you  to  answer  a  single  question  on  the  questionnaire 
and  then,  depending  on  your  answer,  either  return  It  to  us  or  complete 
the  remaining  nine  Items. 

We  would  be  most  grateful  If  you  would  take  a  few  minutes  to  read 
the  definitions  and  descriptions  at  the  beginning  of  the  questionnaire. 
If  you  determine  that  your  organization  does  not  offer  training  of  the 
type  described,  simply  complete  Item  1  and  return  the  questionnaire  in 
the  enclosed  envelope.    If  you  do  offer  training  opportunities  of  the 
type  described,  please  completeTteros  2  through  10  and  return  the 
questionnaire  to  us. 

Your  response  Is  most  Important  to  our  study:  In  order  to  meet  our 
contract  deadline,  we  would  be  most  appreciative  If  the  questionnaire 
could  be  returned  to  us  no  later  than  August  16.    If  you  have  questions 
concerning  the  study,  you  may  contact  me  at  the  address  Indicated  on 
the  return  envelope. 


Sincerely  yours, 


Blaine  R.  Worthen,  Chairman 
Task  Force  on  Research  Training 
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Q^:    QUESTIONNAIRE  FOR  NON-DEGREE  GRANTING  ORGANIZATIONS 


AERA  Research  Training  Study 


i! 


r 
r 

c 

0 
II 
I 

^erIc  I 


In  this  questionnaire,  training  in  educational  research  and  research- 
related  areas  Is  defined  to  Include  training  in  (1)  educational  research . 

(2)  the  design  and  development  of  instructional  products  and  programs , ~ 

(3)  the  dissemination  of  products  and  curricula  to  the  educational 
community,  and  (4)  the  evaluation  of  materials  and  programs,  both  as 
part  of  the  development  effort  and  as  an  ongoing  monitoring  process. 
(These  areas  are  jointly  designated  as  educational  RDOE.) 

This  study  Is  aimed  at  learning  what  training  opportunities  in  the  above- 
mentioned  areas  an  employee  or  member  or  other  interested  person  cnight 
expect  to  encounter  through  your  organization. 

The  kinds  of  training  opportunities  under  Investiqatlon  include  those 
training  vehicles  (such  as  seminars  and  workshops)  which  are  a  scheduled 
anJ  ongoing  part  of  the  overall  activities  of  the  or9ani2ation.'"To'lTiTs 
extent,  they  may  be  considered  "formal"  training  activities,  as  opposed 
to  the  less  structured,  day-to-day  training  of  an  employee  which  takes 
place  naturally  through  his  exposure  to  the  projects  and  people  arounci  him. 

The  questions  which  follow  pertain  only  to  the  sdiedulcd  and  ongpinj 
training  activities  which  your  organization  makes  available.  ^  ' 


Does  your  organization  offer  training  opportunities  uf  the  type  described 

above  in  educational  RDDL  (research,  devclopmrnt. ,  dissemination  am!  evaluation)? 

(Please  check  ONE.) 


YES  □ 


If  you  answered  NO  above,  please  pi-ovide  your  nan«,  title,  and  the  name 
of  your  organizatTon  in  ti»e  space  helcw  and  ret:.jrn  t'li?  que  ti'jnnaire  in 
the  enclosed  envelope*. 

Name  &  Title 


Organization 


If  you  answered  YES  above,  please  complete  items  2  trroiiijh  10  «jnd 
return 'the  questionnaire  in  the  envelope  provided. 


Given  the  types  of  training  activities  listed  in  the  first  column  bf^low, 
please  provide  the  indicated  information  for  each  activity  whirh  yoi.r 
organization  regularly  (e.g.,  weekly,  annually)  oTfers  in  educational 
RDDE  subjects. 


Type  of 
Training  Activity 

Lectures 

Seminars 

Workshops 

Conferences 

Institutes 

Other* 


Freqiienqy  of  Training 
Activity  {e.(i.  >  weekjy| 


Normal  Duration 
of  the  Artivity 


*  See  item  8  for  questions  on  Internship  activities, 
include  internships  here. 


Do  not 


FUase  indicate  >*<ch  apply  to  participation  in  the  RDDC 


3)  the  dissemination  oT  products  and  curricula  to  the  educational 
coimiunity,  and  (4}  the  evaluation  of  materials  and  programs,  both  as 
part  of  the  development  effort  and  as  an  ongoing  monitoring  process. 
(These  areas  are  jointly  designated  as  educational  RODE.) 

This  study  is  aimed  at  learning  Mhat  training  opportunities  in  the  abov*?- 
mentioned  areas  an  employee  or  manber  or  other  interested  person  might 
expect  to  encounter  through  your  organization. 

The  kinds  of  training  opportunities  under  investigation  Dsclude  those 
training  vehicles  (such  as  seminars  and  workshops)  which  arc  a  sdieduled 
and  ongoing  part  of  the  overall  activities  of  the  organization. "'To'TTiTs 
extent,  they  may  be  considered  "formaV*  training  activities,  as  opposed 
to  the  less  structured,  day-to-day  training  of  an  envloyce  which  takes 
place  naturally  through  his  exposure  to  the  projects  and  peo|ile  arouno  him. 

The  questions  which  follow  pertain  only  to  the  scheduled  and  ongoinu 
training  activities  which  your  organization  makes  avaiTahle.  ~ 


Does  your  organization  offer  training  opportunities  uf  the  type  described 
(Ple«rchS?ONrf  ^^^^  ("research,  devtlopmrnt. ,  dissemination  am!  evaluation)? 

YES  □  NO  □ 

If  you  answered  NO  above,  please  pnwide  your  narno,  title,  and  the  name 
Of  your  organization  In  tl»e  space  helcw  and  re^  jrn  the  que-  titmnaire  in 
the  enclosed  envelope. 

Name  &  Title  

OrganlZv^tion     


If  you  answered  YES  abovo,  please  completo  itf?ins  2  throiHih  To  .:nd 
return 'the  questionnaire  in  the  envelope  providijil. 


Given  the  types  of  training  activities  listed  in  the  first  column  br*low 
Kn^n^P.T^*  information  for  each  activity  SlTymM- 

RODE  l5bS?Jsr^^^  annually)  offers  In  educational 

Tuno  «r  r  ^  *   .  .  Normal  Duration 

T   4       11°:.  .  Frequency  of  Training  of  the  Artivitv 

Training  Activity.       Activity  (e.g. ,  weekly!  1?^ JJS 


Lectures 

Seminars 

Horkshops 

Conferences 

Institutes 

Other* 


*  See  item  8  for  questions  on  Internship  activities.    Do  not 
Include  Internships  here. 

Please  indicate  below  the  limitations  which  apply  to  participation  in  the  RDDC 
training  activities  regularly  offered  by  your  organization!  (p  Ssi^^^ 
the  appropriate-column  fdr  each  relevant  activity.)  ifiease  cneck 

T^-inRS^a^r*  ^*  Open  to  Members/  Open  to  Limited  Out-  Open  to  Anyone 
Training  Activity       Employees  Only       side  Participants  Interested 


Lectures 

Seminars 

Workshops 

Confierences 

Institutes 

Other 
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In  general,  is  a  fee  required  for  participation  in  the  training  offr.^ii  to 
your  organization? 

VES  □  NO  □ 

If  YES,  please  estimate  the  average  fee  charged  per  person  $ 

If  NO,  or  if  fees  do  not  fully  cover  expenses  of  the  t.r^^nino,  iiow  •> 
the  program  funded?    (Please  check  ONF,) 

 Primarily  by  your  organization 

 Primarily  by  outside  sources  (Please  specify  ^    ) 

 About  equally  by  organizational  and  outside  support  (N^»ise 

specify   _J 

In  general,  who  are  the  instructors  in  your  organization's  traimno  activities? 
(Please  check  any  which  apply.) 

 Staff  members  from  within  your  organization 

  Persons  from  outside  your  organization,  but  not  culleue  or 

university  factilty   

 Colleye  or  university  faculty  fran  outSJrtP  your  organizrit  ion 


What  is  the  major  objective  of  your  orqanizatlon  in  provifJing  training 
activities?  (Please  chccif  ONE,) 

 infonn  participants  of  new  developments  in  their  field, 

 To  raise  the  aspirations  of  staff  or  members  in  r*»lation  to 

•icw  tediiiiques  or  subjects 

 lo  update  the  training  of  the  participants  in  sublets  v/hlch 

tlicy  iMve  prev^'tusly  SMidied 

 To  pjovfde  in-depth  troinfiig  in  subjects  which  ar«  nn/  to 

the  participants 
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In  general,  who  are  the  instructors  In  your  organization's  trainin<)  activities? 
(Please  check  any  which  apply.) 

 Staff  mRirtiers  from  within  your  oi'g^nization 

 Persons  from  outside  your  organisation,  but  not  cclleue  or 

university  faculty  ~ 

 COlleije  or  university  faculty  frooi  outstrip  your  organijiat  ion 


What  is  the  majm  objective  of  your  orqanuatlon  in  provLJiim  triiininfj 
activities?      '"'(Please  check  ONE.)  r         ^  j 

.  To  inronn  participants  of  new  dcvelopwnts  in  i\\nir  f1eld> 

 To  raise  tho  aspirations  of  staff  or  members  in  relation  to 

-icw  techiviques  or  subjects 

 lo  update  the  training  of  the  participants  in  subjr»cts  vrtiich 

they  have  prev^i^usly  studied 

 To  piovide  in-depth  troln^Hu  in  subjects  wliich  ar«  new  to 

the  participants 

Please  li'it  b(  low  the  topjcs  wiilch  have  bru»n  covered  bv  training  wor|rshops» 
seminars,  etc,  in  your  organization  in  the  p^si  year  or  which  are  planned 
for  presentation  in  Ihe  uwlng  year  fe-j.,  statisticiil  intthods  or  lethniquest 
proposal  dcvelopnert,  research  manaoenieiit) •    Also  indicate  for  each  topic 
whether  it  was  (or  will  be)  presented  as  .i  lecture  (L),  seminar'  (SK 
woricshop  (M),  coiiferonco  (C),  or  Institute  (I). 


Does  your  organization  wo»lc  CQoperativelv  with  colloyes  or  universities  to 
offer  fonnal  Internship  opportunities  foi  degree  or  postdoctoral  s.tudent$ 
in  educational  research  and  research-related  areas?  . 

VESQ  NOn 

If  YES,  with  what  colleges  and  universities  have  internship  armngements 
been  rfiade  in  the  past  throe  years? 


If  your  organization  einploys  personnel  in  educational  research  and  roscarch- 

to  those  who  w^sh  to  talce  credit  courses  In  an  accredited  college  or 
university.   (Please  checic  any  item  which  applies.) 

—  Tuition  for  employees  is  paid  by  your  organization  only  for 

T  i^4^^^V^^  directly  related  to  their  work  in  your  organization. 

—  Tuition  for  employees  is  paid  by  your  organization  for  any 

course  in  the  employee's  degree  program. 
 No  tuition  for  employees  is  paid  by  your  organization. 

 Employees  have  time  off  with  p^y  to  attend  classes  directly 

related  to  their  work  in  your  organization. 
 Employees  have  time  off  with  pay  to  attend  classes  in  their 

degree  program. 

—  No  time  off  with  p$jf  is  given  for  class  attendance  by  einiloyees. 

In  the  space  be1ow»  please  give  the  name  and  title  of  the  person  within 

your  oraanization  who  may  be  contacted  by  those  wishing  additional  inroraiatlon 

concerning  research  traininjjjctivities.  •  •nurmoi.iun 

Name  I  Title  ^J^jir  ' 

Organization     
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ameRican  eoucaTionaL 
ReseancH  associaiion 

FOLLOW-UP  LEHER 

August  11 »  1971 


Dear  Colleague: 

Some  weeks  ago  we  sent  you  a  brief  questionnaire  to 
collect  data  concerning  training  programs  In  educational  research 
and  research-related  areas.   We  regret  having  to  bother  you  again, 
but  to  date  we  have  not  received  a  response  from  you. 

We  would  be  most  grateful  if  you  would  take  a  few 
minutes  to  read  the  definitions  at  the  beginning  of  the  enclosed 
questionnaire.    If,  on  the  basis  of  those  definitions,  you 
determine  that  your  Institution  does  not  offer  a  graduate  program 
In  educational  research  and  research- riTa ted  areas,  simply  check 
Item  1  and  return  the  questionnaire  In  the  enclosed  envelope. 

If  you  do  have  a  graduate  educational  research  or  research- 
related  program  according  to  our  definitions,  please  forward  the 
questionnaire  and  this  cover  letter  to  the  director  of  the  program 
or  to  the  faculty  member  who  is  best  able  to  describe  the  program. 
He  should  then  complete  the  questionnaire.  Including  the  Informa- 
tion In  Item  11  concerning  his  position,  and  return  the 
questionnaire  to  us. 

Your  response  Is  most  Important  to  our  stu4y,  and  the  value 
of  the  Investigation  will  be  seriously  reduced  unless  a  much  higher 
percentage  of  responses  Is  received.   We  would  like  to  request 
again  that  you  complete  and  return  the  questionnaire. 

We  will  be  grateful  for  your  cooperation  and  assistance. 

Sincerely, 

Blaine  R.  Worth en,  Chairman 
Task  Force  on  Research  Training 

BRW/lg 
Enclosures 


1126  SIXTEENTH  STREET.  N.W.   •   WASHINGTON,  D.  0.  20036  •  202/223-9485 
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amenican  eoucaTioruiL 
ReseaRCH  assoc  lai  k  -i  ) 

FOLLOW-UP  LEHER 

August  26,  1971 


Dear  Colleague: 

Some  weeks  ago  we  sent  you  a  brief  questionnaire  to 
collect  data  concerning  training  activities  In  educational 
research  and  research-related  areas.   We  regret  having  to 
bother  you  again,  but  to  date  we  have  not  received  a  response 
from  you. 

We  would  be  most  grateful  If  you  would  take  a  few 
minutes  to  read  the  definitions  at  the  beginning  of  the  enclosed 
questionnaire.    If,  on  the  basis  of  those  definitions,  you 
determine  that  your  organization  does  not  offer  training  of  the 
type  described,  simply  fill  In  Item  1  anH'  return  the  question- 
naire In  the  enclosed  envelope.    If  you  do  of fer  training 
opportunities  of  the  type  described,  please  complete  Items  2 
through  10  and  return  the  questionnaire  to  us. 

Your  response  Is  most  Important  to  our  study,  and  the 
value  of  the  Investigation  will  be  seriously  reduced  unless  a 
much  higher  percentage  of  responses  Is  received.   We  would  like 
to  request  again  that  you  complete  and  return  the  questionnaire. 

We  will  be  grateful  for  your  cooperation  and  assistance. 

Sincerely, 

Blaine  R.  Worthen,  Chairman 
Task  Force  on  Research  Training 

BRW/mb 
Enclosures 


1126  SIXTEENTH  STREET.  N.W.   •   WASHINGTON.  D.  C.  20036  •  202/223-9485 
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Some  Arguments  Against  Considering  Retraining  of 
Unemployed  Doctorates  In  Unrelated  Fields  as;a  Major  Source  of  Supply 

of  Educational  RDDE  Personnel 


It  was  suggested  by  the  deans  surveyed  by  Phi  Delta  Kappa  that 
current  educational  RDDE  tralnl/ig  programs  be  expanded  to  meet  the  need 
for  more  trained  RDDE  personnel  (Robertson  &  Sistler,  1971,  p.  63).  It 
has  been  proposed  by  other  persons,  however,  that  unemployed  Ph.D.'s  from 
fields  outside  education  are  a  prime  recruitment  group  to  meet  the  need 
for  trained  personnel  In  educational  research,  development,  diffusion,  and 
evaluation.   Those  who  support  the  idea  of  re-training  unemployed  Ph.D.'s 
argue  (1)  re-training  would  provide  the  trained  manpower  needed  without 
expanding  an  already  too  large  capacity  for  training  Ph.D.'s,  (2)  re- 
training the  already  trained  would  be  faster  and  more  economical ,  and 
(3)  re-training  trained  persons  who  are  without  jobs  is  a  better  use  of 
humarf  resources  than  upgrading  the  training  of  persons  who  are  alreac(y 
employed. 

A  study  conducted  by  Dean  and  others  identified  several  barriers  to 
Implementing  that  strategy,  however.   For  one  thing,  no  large-scale 
retraining  efforts  are  In  operation  now,  and  none  are  anticipated  for  the 
near  future.  The  few  retraining  programs  now  in  existence  are  very  expen- 
sive; there  is  a  one  year  master's  level  retraining  program  at  Stanford 
University  which  costs  over  $10,000  per  student  to  conduct.   It  is  unknown 
at  the  present  time  where  the  funding  would  come  from  for  such  expensive 
programs.   Also,  there  are  mny  unanswered  questions  about  ultimate 
placement  for  these  retrained  Ph.D.'s.   Will  there  be  a  job  for  a  retrained 
person  who  has  previously  specialized  in  another  field?   (Dean,  et  al_.,  1"971, 
p.  17) 
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Even  If  these  barriers  could  be  eliminated,  there  would  still  be  the 
question  of  whether  a  person  who  has  devoted  the  time  necessary  to  obtain 
a  Ph.D.  in  one  field  would  be  willing  to  give  that  up  and  start  over  again, 
in  another  field  (and  perhaps  end  up  in  a  lower  status  position,  at  that; 
such  would  be  the  case  if  a  Ph.D.  entered  the  retraining  program  at 
Stanford) . 

For  the  sake  of  efficiency  (so  that  the  persons  being  retrained  would 
not  have  to  spend  as  much  time  in  a  training  program  as  would  persons 

a 

entering  graduate  work  for  the  first  time),  the  retraining  programs  should 
be  tailored  to  fit  personnel  previously  in  specific  fields,  e.g.,  engineers. 
Designing  and  developing  a  tailored  retraining  program  for  engineers 
would  be  expensive  and  time-consuming,  but  it  could  be  done.   The  question 
is,  would  the  expense  and  the  time  be  worth  it?   Would  the  job  market 
have  corrected  itself  so  that  supply  and  demand  would  have  returned  to  a 
state  of  balance  in  a  particular  field  such  as  engineering  by  the  time  a 
retraining  program  for  engineers  could  be  designed?  There  is  no  way  to 
predict  that.   Even  if  a  retraining  program  could  be  designed,  would  it 
be  very  widely  used  once  it  was  developed?   Deisigning  the  program  is  but 
a  small  part  of  the  work;  It  must  also  be  disseminated  and  adopted,  and 
the  dif fusion/ adoption  pattern  as  used  now  Is  not  very  impressive. 

Therefore,  as  a  result  of  these  barriers  and  questions.  It  is  the 
opinion- of  the  author  that  retraining  unemployed  Ph.D. 's  from  fields  other 
than  education  is  not  one  of  the  better  strategies  for  preparing  persons, 
to  work  In  educational  RDDE. 
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